A Data-Driven Computational Semiotics: The

A Data-Driven Computational Semiotics: The Semantic Vector Space of Magritte’s Artworks Semiotica, special issue Meaningful Data, 2018

Jean-François Chartier1, Davide Pulizzotto1, Louis Chartrand1, Jean-Guy Meunier1 1

Laboratory of Cognitive Analysis of Information, UQAM, Montreal, Canada

Abstract: The rise of big digital data is changing the framework within which linguists, sociologists, anthropologists and other researchers are working. Semiotics is not spared by this paradigm shift. A data-driven computational semiotics is the study with an intensive use of computational methods of patterns in human-created contents related to semiotic phenomena. One of the most promising frameworks in this research program is the Semantic Vector Space (SVS) models and their methods. The objective of this article is to contribute to the exploration of the SVS for a computational semiotics by showing what types of semiotic analysis can be accomplished within this framework. The study is applied to a unique body of digitized artworks. We conducted three short experiments in which we explore three types of semiotic analysis: paradigmatic analysis, componential analysis and topic modelling analysis. The results reported show that the SVS constitutes a powerful framework within which various types of semiotic analysis can be carried out. Keywords: Computational Semiotics; Semantic Vector Space; Data-Driven; Paradigmatic Analysis; Component Analysis; Topic Analysis; Artwork Mining; René Magritte. 1. Introduction The rise of big digital data brings new opportunities in various research areas of humanities and social sciences. Today, thanks to corpora such as the Google N-Gram, a multi-million book corpus covering a period of more than 500 years, anthropologists can study the evolution of the human culture while using an observational scale that was previously inaccessible (Michel et al. 2011). Linguists can track linguistic shifts, as they evolve in a language over centuries (Hamilton, Leskovec, and Jurafsky 2016). With so many classical scholarship digitization projects, historiography is also becoming a large-scale empirical analysis enterprise (Mimno 2012). The social world is submerged with digital footprints from the Web that represent invaluable big data sets for sociological analysis (Evans and Aceves 2016). The rise of big digital data brings also new challenges, in particular, the ones that come with the computational frameworks needed to study these data. 1

We hear more and more about the "data revolution" and what is now called the new "dataintensive" or "data-driven" approach (Kell and Oliver 2004; Kelling et al. 2009; Kitchin 2014; Pankratius et al. 2016). A data-driven approach is a computer-based exploratory framework that uses data mining algorithms to discover in large dataset regularities or patterns of interest. In this framework, algorithms are computational enhancements for the discoveries of these patterns. As noted by many, in just a decade, digital (big) data and algorithms have changed the framework within which linguists, sociologists, anthropologists, semioticians and other researchers do their work. This entails a new relationship between researchers and the empirical evidence they uncover (Rastier 2011:11). A computational framework based on an intensive use of data mining algorithms is more than ever a critical issue in the humanities and social sciences. Semiotics is not spared by this paradigm shift. A data-driven computational semiotics has been emerging for some years. 1.1. A Data-Driven Computational Semiotics Semiotics is defined as the study of signs and of all systems and processes concerning semiosis. Computational semiotics may be considered as a new branch of semiotics. However, its definition is not easy to grasp. One can find at least three research programs that claim to belong to computational semiotics (Meunier 2017; Tanaka-Ishii 2015). In one of these programs, computational semiotics means that computation is a kind of semiotic theory itself. In this view, one would say for instance that a computer is a semiotic system: “[…] in order to be meaningful, computers ought to be semiotic machines. […] To say that the computer is a semiotic machine means to realize that what counts in the functioning of such machines are not electrons (and in the future, light or quanta or organic matter), but information and meaning expressed in semiotic forms, in programs, in particular, or, more recently, in apps.” (Nadin 2011:172)

This definition can be found in several research programs of artificial intelligence (Floridi 1999; Meunier 1989). In a second research program, computational semiotics is about human-machine interaction and design (De Souza 2005). This research program includes many subfields like the semiotics of programming language and software engineering (Tanaka-Ishii 2010). A third research program refers to computer-based methodologies. Here, computational semiotics means studying semiotic phenomena in a human-created content — all kinds of content, from text to artwork — with computational models and methods. In this research program, we develop computational models and methods that can be manipulated by a computer and applied to datasets in order to mine and extract important mathematical patterns correlated with phenomena of semiosis. This kind of work remains however very marginal in the semiotic community. The relation between those mathematical patterns in data and semiosis still remains a controversial open issue. Several semioticians will reject the hypothesis that semiotic processes could be modelled in a computational way (Meunier 2014). 2

Nevertheless, some of the most promising works in this program are those about the Semantic Vector Space (SVS) models and their methods (Leopold 2005; Mehler 2003; Neuman, Cohen, and Assaf 2015; Rieger 1999; Sahlgren 2006). SVS models are a mathematical modelling of combinatorial behavior of signs correlated with different semiotic phenomena, and a collection of algorithm-based methods with which one seeks to discover these patterns in large corpora. Currently, the main leaders in this research program of computational semiotics are not semioticians but computer scientists specialized in natural language processing, data mining and information retrieval (Turney and Pantel 2010; Van Rijsbergen 2004; Widdows 2004). The benefits of this work are, however, important for computational semiotics. They have shown that the SVS allows one to discover combinatorial patterns from which one can infer complex meaning structures like paradigmatic relation, componential relation, metaphor, analogy, topics and so on. 1.2. Aim of This Research The aim of this article is to explore the contribution of SVS for a computational semiotics research program. How can SVS constitute a theoretical and methodological framework from which to build a data-driven computational semiotics? What is the productivity of this framework for semiotics, in other words, what types of semiotic analysis can we conduct based on that framework? In order to provide answers to these questions, we first present the theoretical framework of the SVS, its main parameters and the main operators of its semantic inference engine. The SVS is usually presented from a linguistic or information retrieval perspective and applied on text analysis. In this study, we try to separate this framework from this particular perspective. To achieve this, we apply the SVS and its algorithms to an unprecedented corpus composed of the surrealist painter René Magritte’s artworks. To our knowledge, this is the first time that the SVS is applied as we have applied it to the analysis of the type of sign we find in this corpus, that is to say, iconic visual signs. We conduct three short experiments on this corpus. Each aims to explore a kind of semiotic analysis applied to the corpus of Magritte’s artworks with the algorithms of the SVS. The first experiment is a paradigmatic analysis. It is based on a semantic comparison algorithm and aims to discover silences in the studied corpus. The second experiment is a componential analysis. It is based on a semantic decomposition algorithm and aims to discover how some iconic visual signs are composed in the studied corpus. Finally, the third experiment is a topic modelling analysis. It is based on a partitioning algorithm and aims to discover the topics around which Magritte’s artworks are organized. These three experiments are not intended to show all the semiotic analyzes one can accomplish with the help of the SVS framework. We rather wish to give a partial overview of what we think constitutes a very promising new branch of semiotics. As a matter of fact, those experiments do not only help to answer our questions, they will also raise several issues for semiotics.

3

2. Theoretical Framework: The Semantic Vector Space Model The semantic vector space model (SVS) is both a computational model and algorithmicbased methods for the inductive discovery from combinatorial patterns of signs in a corpus (usually textual, but not exclusively) of meaning structures. The model is based on a spatial metaphor, whereas the methods are based on a distributional hypothesis (Gärdenfors 2014; Landauer and Dumais 1997; Lemaire and Denhière 2006; Sahlgren 2006; Turney and Pantel 2010; Widdows 2004). Its theoretical sources are numerous and multidisciplinary, including computer sciences, psychology and linguistics. Originally, it is a synthesis between, firstly, the spatial metaphor operationalized in the differential semantics of Osgood (Osgood 1952, 1964; Osgood, Suci, and Tannenbaum 1957) and, secondly, the distributional hypothesis of structural linguistics developed by Harris (Harris 1951, 1954). 2.1. The Vector Space Metaphor The spatial metaphor of the SVS model was introduced in Osgood’s psycholinguistic work, but some have suggested that this metaphor goes back as far as Saussurre’s classic writing and the concept of value (Sahlgren 2008). Actually, Osgood wanted to resume a Saussurian principle, but by operationalizing it in a more formal framework. He sought to measure and represent mathematically the difference of linguistic sign connotations observed among different subjects. To do so, these connotations were collected through a conceptual priming task. For example, subjects were invited to evaluate on a seven-level scale the connotation of the word "FATHER" according to the three following dimensions "happy vs sad", "hard vs soft" and "slow vs fast" (see Figure 1) (Osgood et al. 1957:26). The responses to this priming were then encoded in vectors and projected into a vector space called metaphorically by Osgood a "semantic space". Osgood’s differential semantic model consisted in analyzing the distance between word vectors in this semantic space. According to Osgood, this model allowed an objective measure of sign connotative meaning.

Figure 1: Example used in Osgood’s experiments to measure the connotative judgments of the word "FATHER".

This spatial metaphor is coherent with a structural approach of meaning, as opposed to a referential one. The assumption underlying this spatial metaphor was to conjecture isomorphism between the vector space and the semantic space: « Since the positions checked on the scales constitute the coordinates of the [sign]’s location in semantic space, we assume that the coordinates in the measurement space are functionally equivalent with the components of the representational mediation process associated with this [sign]. This, then, is one rationale by which the semantic differential,

4

as a technique of measurement, can be considered as an index of meaning. » (Osgood et al. 1957:30)1

In other words, there is a correlation between some measurable mathematical properties of the vector space and some non-directly observable properties of meaning structure of signs encoded in that space. Therefore, the analysis of the first – with algebraic, topological and geometrical calculi – should enable the inference of the second. This assumption is at the root of contemporary SVS models. See for instance Gärdenfors for whom : « [the meaning] can be described as organized in abstract spatial structures that are expressed in terms of dimensions, distances, regions, and other geometric notions […] » (Gärdenfors 2014:22)

The fundamental theoretical elements that constitute the conceptual framework of SVS were almost all already present in Osgood’s work. What will differentiate the contemporary SVS from Osgood’s classical work is the integration of the distributional hypothesis in that framework. The priming technique used by Osgood to construct a semantic space is substituted in the contemporary SVS framework by statistical analysis techniques that enable the vector space induction from sign combinatorial patterns computed in a corpus. 2.2. The Distributional Method The core concepts of Harris’s distributional method are those of environment and distribution. The concept of environment defines the analysis unit of sign context or neighborhood in a corpus and determines the combinatorial relations that a sign has with other signs of a corpus. This analysis unit can take several forms depending on the corpus. On a textual corpus, it may correspond to a grammatical unit such as collocation, sentence and paragraph, but it can also be a more arbitrary unit like a window of words or be linked to pragmatic criteria like speaking turns. The concept of distribution is derived from the concept of environment. For Harris, all sign combinations observed in a corpus, that is to say the set of all its environments, forms its distribution. According to the distributional hypothesis, signs characterized by equivalent distributions would themselves be " functionally" equivalent (Harris 1951:16). Although Harris has focused on syntax, according to him, this distributional hypothesis could also be applied to meaning in a corpus. The meaning of a word in a corpus is correlated with its combinatorial patterns with other words in the corpus. Words with similar co-occurrence distributions would also have a similar meaning: « The fact that, for example, not every adjective occurs with every noun can be used as a measure of meaning difference. For it is not merely that different members of the one class have different selections of members of the other class with which they are actually found. More than that: if we consider words or morphemes A and B to be more different

1

Osgood uses the words "word", "concept", "sign" in an undifferentiated way. In order to avoid theoretical ambiguities, in this quotation the English word "concept" has been replaced by the word "sign". This does not change the meaning of the sentence. 5

than A and C, then we will often find that the distributions of A and B are more different than the distributions of A and C. In other words, difference in meaning correlates with difference in distribution. » (Harris 1954:156)

This quotation echoes another famous one from the linguist Firth who used to say that "You shall know a word by the company it keeps » (Firth 1957:179), which means that word meaning comparison can be reduced to word co-occurrence patterns comparison. This distributional hypothesis represents the cornerstone of the contemporary SVS methodology. Its principle is surprisingly simple : words, morphemes, syntagms, and potentially other signs expressing similar meaning are generally used in similar environments, hence one can induce which signs express similar meaning by comparing their combinatorial patterns in a corpus. 2.3. The Semantic Vector Space Parameters There are several types of SVS, including explicit semantic spaces (Lund and Burgess 1996; Rieger 1981), latent semantic spaces (Landauer, Foltz, and Laham 1998), spaces constructed from random indexing (Sahlgren 2005), word embedding based semantic spaces (Mikolov, Yih, and Zweig 2013), tensor-based spaces (Baroni and Lenci 2010) and we can also add several probabilistic models (Blei and Lafferty 2009; Griffiths, Steyvers, and Tenenbaum 2007). All are based on the same fundamental assumptions mentioned above, but operationalized differently. The model that we present is a three parameters model < 𝕍, ℂ, 𝕎 >, with a vocabulary 𝕍, a set of contexts ℂ and a matrix of combinatorial relations 𝕎. The first parameter is the vocabulary of the studied corpus. It is denoted by 𝕍 = {𝑣1 … , 𝑣𝑚 } and represents a set of selected signs to represent the content of the studied corpus. It is for instance a set of words present in a textual corpus. The vocabulary forms the vector base of the SVS and determines its dimensionality. The second parameter, denoted by ℂ = {𝑐1 … 𝑐𝑛 }, represents the set of contexts (or to use Harris’s terms, the set of environments) that make up a corpus. A context 𝑐𝑘 = {𝑣i : 𝑣i ∈ 𝕍} corresponds to a set of signs co-present in the same segment of a corpus. For instance, in a textual corpus, a context could be the set of words co-present in the same sentence. The segmentation of a corpus into its different contexts is an operation that will determine what type of combinatorial regularity between signs the SVS will model. m×m

The third parameter, denoted 𝕎 = [𝑤𝑖𝑗 ] ∈ ℝ𝑚 is the matrix of combinatorial relations between the signs of the vocabulary 𝕍, in which 𝑤𝑖𝑗 corresponds to the value of a statistical association coefficient between two signs 𝑣i and 𝑣j . This association coefficient expresses quantitatively the strength of the combinatorial relation between two signs. Figure 2 illustrates schematically this matrix.

6

sign1

sign2

…

signm

sign1

w11

w12

…

w1m

sign2

w21

w22

…

w2m

⋮

⋮

⋮

⋱

⋮

signm

wm1

wm2

…

wmm

Figure 2: Matrix of combinatorial relations between signs.

A row vector 𝐯 = (𝑤1 , … 𝑤𝑚 ) of the matrix encodes the combinatorial pattern of a sign in the studied corpus or, to use Harris’s terms, this vector models the “distribution” of this sign in the corpus. These vectors are the coordinates of the signs in the SVS in a manner analogous to what Osgood did, except that rather than representing the results of a semantic priming task, the vectors of 𝐯 ∈ 𝕎 represent combinatorial patterns in a corpus. According to Rieger, this matrix of combinatorial patterns can be interpreted as a “syntagmatic abstraction” of the relations between signs (Rieger 1992). In this matrix, the value of wij can be seen as the quantification of the strength of the syntagmatic relation between the signs vi and vj in a corpus. The vector 𝐯𝐢 can be seen as the syntagmatic signature vector of this sign. The strength of the syntagmatic relation between two signs must be understood in a very reductive sense: it is a methodological reduction of syntagmatic relations (collocation, phrasal verbs, etc.) to co-occurrence patterns (Bordag and Heyer 2007; Sahlgren 2008; Schütze and Pedersen 1993). Nevertheless, this methodological reduction represents one of the most reliable theoretical foundations of distributional linguistics (Dunning 1993; Manning and Schütze 1999:5). The rationale for sign combinatorial patterns calculation is co-occurrence frequency, that is the number of times two signs are co-present within the same contexts. The assumption is that the more frequent two signs are co-present in the corpus, the stronger the relation is between them. In our model, combinatorial patterns between signs are calculated using the information gain (IG) coefficient2. It is calculated as follows: 𝐼𝐺(𝑣𝑖 , 𝑣𝑗 ) = 𝑤𝑖𝑗 = ∑2𝑖=1 ∑2𝑗=1 (

𝑛𝑖𝑗 𝑁

log 2 (𝑛

𝑁×𝑛𝑖𝑗

),

𝑖1 +𝑛𝑖0 )×(𝑛1𝑖 +𝑛0𝑗 )

where 𝑛11 is the number of contexts where the signs 𝑣𝑖 and 𝑣𝑗 are co-presents, 𝑛10 is the number of contexts where appears the sign 𝑣𝑖 but not the sign 𝑣𝑗 , 𝑛01 is the number of contexts where does not appear 𝑣𝑖 but where 𝑣𝑗 does, 𝑛00 is the number of contexts where the signs 𝑣𝑖 and 𝑣𝑗 do not appear et N=|ℂ| is the total number of contexts in the corpus.

It is sometimes called “mutual information”. Note that it is very different than the pointwise mutual information commonly used in the construction of a semantic vector space, but that also comes with many issues, see (Bouma 2009) 2

7

IG is a critical feature of our SVS model. It is a probabilistic measure of the statistical dependence between two elements. It is also one of the most suited coefficients for cooccurrence analysis since it has many advantages (like a smoothing effect) over a raw frequency calculation (Bouma 2009; Manning and Schütze 1999:178). In our framework, this coefficient can be interpreted as follows. A zero gain information between two signs means that they have statistically independent combinatorial behaviors. Conversely, an IG of 1.0 between two signs means totally dependent combinatorial behaviors in a corpus: the presence of the former in a context always implies the presence of the latter and the absence of one always implies the absence of the other (and vice versa). Empirically, most of combinatorial patterns are generally somewhere between these two limits. 2.4. The Semantic Inference Engine The ability of a SVS model to infer certain meaning structures is grounded in this algebraic formalism presented above. A strong assumption upon which relies this model consists in conjecturing that several inferences about meaning structures can be based on algebraic calculi applied to the elements of a SVS. These vector calculi are currently the subject of several research programs, notably under the names “vector symbolic architecture” (Widdows and Cohen 2014) and “conceptual space architecture” (Gärdenfors 2014). Three vector calculi are particularly important. The first one is the cosine. It allows sign semantic comparison. The second one is the vector addition. It allows the sign semantic composition. The third one is the subtraction of the orthogonal complement, which allows sign semantic negation. These operations are defined in Table 1 (these definitions assume unit normed vectors). Table 1: Basic inferences on a semantic vector space. Inference

Vector calculus

Semantic comparison

Cosine

Semantic composition

Addition

Semantic negation

Orthogonal Complement Subtraction

Operation

Output 𝐜𝐨𝐬(𝐮, 𝐯) =

𝐮 ∙ 𝐯 = (𝑢1 𝑣1 + … + 𝑢𝑚 𝑣𝑚 ) 𝐀𝐃𝐃(𝐮, 𝐯) = 𝐮 + 𝐯 = (𝑢1 + 𝑣1 , … , 𝑢𝑚 + 𝑣𝑚 ) 𝐃𝐈𝐅𝐅(𝐮, 𝐯) = 𝐮 − (𝐮 ∙ 𝐯)𝐯 = (𝑢1 − 𝜆𝑣1 , … , 𝑢𝑚 − 𝜆𝑣𝑚 )

𝜃∈ℝ

𝐰 ∈ ℝ𝒎

𝐰 ∈ ℝ𝒎

2.4.1. The Cosine Comparator A first basic operation on an SVS, which we denote by 𝐜𝐨𝐬(𝐮, 𝐯) ∈ ℝ, is the calculation of the cosine metric between two vectors. It takes as arguments two vectors and produces a scalar representing the angle between the two vectors. The cosine is the reduction of a semantic similarity comparative inference between two signs to the computation of a metric 8

in the SVS. A 𝐜𝐨𝐬(𝐮, 𝐯) = 1 means perfectly collinear vectors and 𝐜𝐨𝐬(𝐮, 𝐯) = 0 means that the vectors are orthogonal. In an SVS, the semantic similarity is correlated with the spatial proximity. The more two signs are close to each other in the SVS, the more they are semantically similar. This inference is easily explained by the distributional hypothesis: the spatial distance between two signs is small when the signs have similar combinatorial patterns in a corpus and signs having similar patterns tend to share meaning. Other metrics can calculate this semantic similarity inference (Kiela and Clark 2014), but beyond technical differences they are all based on the same methodological reduction, that is, they reduce the semantic similarity to a very particular kind of mathematical similarity. These metrics are measurements of the degree of substitutability or permutability of two signs in a corpus: two signs are semantically similar if they have similar combinatorial patterns, that is, they are both permutable to each other in a corpus (Burgess 2000; Burgess, Livesay, and Lund 1998; Lund and Burgess 1996). This mathematical concept of permutation is interpreted in the SVS framework as a paradigmatic relation between signs in a corpus, in a sense very similar to the Saussurian structural linguistics (Bordag and Heyer 2007; Sahlgren 2008). Rieger used to call the function of this metric a “paradigmatic abstraction” of relations between signs (Rieger 1989, 1992). This is a fundamental inference one can make with the SVS framework. 2.4.2. The Vector Additive Composition A second basic operation of the SVS framework, denoted by 𝐀𝐃𝐃(𝐮, 𝐯) ∈ ℝ𝒎 , is vector addition. It takes as arguments two vectors u and v and produces a third one w which is the element-wise addition 𝐰 = (𝑢1 + 𝑣1 , … , 𝑢𝑚 + 𝑣𝑚 ). The operation 𝐀𝐃𝐃(𝐮, 𝐯) is a componential inference reduced to a linear addition of vectors. The representation in an SVS of the componential meaning of two signs u and v corresponds to the addition of their respective vectors. In other words, the vector representation of a componential structure 𝐰 corresponds to the addition of the sum of the syntagmatic signature vector of its components 𝐮 and 𝐯. The semantic composition is a complex semiotic phenomenon and several algebraic calculations have been proposed to infer it, including projection operators, tensor multiplication and others (Mitchell and Lapata 2010; Widdows 2008). 𝐀𝐃𝐃(𝐮, 𝐯) is its simplest algebraic operationalization, but it comes with important consequences. For instance, the operation is commutative, which means that 𝐮 + 𝐯 = 𝐯 + 𝐮. This can be an issue for the inference of componential relation where element order in the structure is important. 2.4.3. The Subtraction of the Orthogonal Complement Vector The semantic negation, denoted 𝐃𝐈𝐅𝐅(𝐮, 𝐯) ∈ ℝ𝒎 , is another fundamental operator of the SVS framework. It is another form of componential inference, but unlike 𝐀𝐃𝐃(𝐮, 𝐯) which enables componential inference from components to structure, 𝐃𝐈𝐅𝐅(𝐮, 𝐯) allows 9

componential inference from structure to its components. It is Widdows who introduced this inference operator. He summarizes its rational in the following way: « Vector negation is based on the intuition that unrelated meanings should be orthogonal to one another, which is to say that they should have no features in common at all. Thus vector negation generates a ‘meaning vector’ which is completely orthogonal to the negated term. » (Widdows 2003:137)

The semantic negation operator 𝐃𝐈𝐅𝐅(𝐮, 𝐯) is a projection of 𝐮 in the subspace orthogonal to 𝐯 by subtracting from 𝐮 its orthogonal complement in order to obtain a new vector 𝐰 = 𝐮 − (𝐮 ∙ 𝐯)𝐯 representing exactly the meaning of 𝐮 independent of 𝐯, which means that 𝐜𝐨𝐬(𝐰, 𝐯) = 0, but that 𝐜𝐨𝐬(𝐰, 𝐮) will tend to be high. In other words, to decompose the syntagmatic signature vector of a sign 𝐮 by another one 𝐯, one must subtract from 𝐮 the cooccurrence relations it shares with 𝐯. 2.4.4. Complex Inferences The cosine, the vector addition and the subtraction of the orthogonal complement form the basic vector operations of a semantic inference engine. These basic operations can also be combined to form complex algorithms that can produce different semiotic analysis of signs encoded in a SVS. Many of those complex inferences have been studied recently, especially metaphorical relation (Shutova 2010), analogy (Mikolov et al. 2013), hypernymy (Santus, Lu, et al. 2014), hyponymy (Erk 2009) and antonymy (Lu 2015). This inference engine is particularly interesting for a data-driven approach such as the one we want to propose for a computational semiotics. As Widdows and Cohen pointed out, vector calculi on SVS “support fast, approximate but robust inference and hypothesis generation” (Widdows and Cohen 2014:141). In fact, these algebraic calculi are all based on very reductive inductive biases such as linearity and commutativity. However, this methodological reduction enables large-scale empirical analysis of very complex semiotic phenomena that would otherwise be very difficult to study on that scale. 3. Experiments on René Magritte’s Artworks Semantic Indexing In order to show how the semantic inference engine of the SVS framework functions, we have conducted three short exploratory experimentations where we have implemented various algorithms based on the vector calculi defined earlier. As noted in the introduction, our aim is to demonstrate the relevance of this framework for a data-driven computational semiotics. Our experiments are conducted on a unique corpus constituted of René Magritte’s artworks semantic indexing. We adopt a genuine data-driven exploratory research design here, where we let our data mining algorithms recognize and retrieve specific combinatorial patterns from this corpus. We seek to show what kind of semiotic analysis these algorithms lead to and perhaps even challenge the scholars of Magritte’s artworks. Before presenting those three experiments, we first present in details the characteristics of the studied corpus as well as its pre-processing.

10

3.1. The Semantic Indexing of Magritte’s Artworks In digital humanities, several kinds of large digital corpora are created: textual corpora, social network corpora, geolocation corpora, etc. The creation of large digitized corpora of artworks is in the line with this ongoing effort. In recent years, several unique corpora composed of thousands of digitized artworks have been created for analytical purposes (Arora 2012; Carneiro et al. 2012; Liu et al. 2007; Zhang, Islam, and Lu 2012). A reference corpus is, for instance, the PAINTING-91 (Khan et al. 2014). This corpus consists of 4,266 digitized images of paintings made by 91 different artists. There are 49 artworks of Francisco de Goya, 39 of Raphael and 50 of Picasso. Another corpus recently created and much more voluminous is the Wikiart3. It contains 81,449 digitized images of artworks of 1,119 different artists, classified into 27 different styles (Saleh and Elgammal 2015). There are more and more initiatives of this kind, see for example (Arora 2012; Carneiro et al. 2012; Crowley and Zisserman 2014; Graham et al. 2010; Johnson Jr et al. 2008; Khan et al. 2014; Li and Wang 2004; Lombardi 2005; Shamir 2012, 2015; Shamir and Tarakhovsky 2012; Shen 2009; Stork 2009; Zujovic et al. 2009). However, the algorithm-based analysis of those digitized artworks corpora remains a very confined research area. It is mainly computer scientists who explore these corpora using data mining, machine learning and pattern recognition algorithms in order to predict authors and artistic styles. Khan and his colleagues, for example used different algorithms from the artificial vision field to automatically recognize authors and styles of artworks, in their analysis of the PAINTING-91 corpus. Saleh and Elgammal analyzed which metrics optimize an automatic recommendation system based on the similarity of plastic features of artworks (Saleh and Elgammal 2015). Li and Wang reconstructed the stroke signature of classical Chinese painters using machine learning algorithms (a hidden Markov model). They also showed that these signatures are sufficiently specific to allow author prediction of artworks (Li and Wang 2004). In semiotics, there is still very little research conducted on the large-scale algorithmic-based analysis of these corpora. This lack of interest from semioticians for the computational study of these corpora can be explained in several ways. A major issue hindering research in this area is what is called the “semantic gap” of the computational analysis of digitized image corpora, especially digitized pictorial artwork corpora (Hare et al. 2006; Liu et al. 2007; Smeulders et al. 2000). Indeed, image mining in contrast to text mining comes with a particular problem: unlike texts, where one can induce meaning structure from word combination patterns (co-occurrences), there is no such patterns in images that an algorithm can recognize and extract in order to induce meaning structures in images. In fact, in most cases, algorithmic-based image analysis is achieved only with the analysis of low-level plastic features of artworks such as texture, pigmentation, geometrical forms, paint strokes, color tones, spaces, lines and others. Mathematical encoding of artworks with these plastic features allow some similarity analyzes (Smeulders et al. 2000), but it does not allow an

3

https://www.wikiart.org/ 11

algorithm to infer from these low-level features a so-called “high-level” analysis of meaning structure in artworks. However, for semioticians, plastic analysis is complementary but generally insufficient, they are also interested in the iconic analysis of these artworks. In fact, in some experimental settings, such as the SVRC scientific competition4, state-ofthe-art machine learning algorithms have already crossed this semantic gap and have already allowed complex iconic image analysis. This is the case of several artificial deep convolutional neural networks (Krizhevsky, Sutskever, and Hinton 2012). Some of these models are even better than humans in task like image semantic categorization (He et al. 2016). However, the generalization of these models to the iconic analysis of digitized pictorial artworks is still an open research topic (Bar, Levy, and Wolf 2014). And so far, the algorithm-based iconic analysis of digitized pictorial artworks is carried out thanks to the introduction of an intermediate (human) task of semantic indexing. Artwork semantic indexing involves attributing to artworks semantic annotations, that is, a list of semantic descriptors. The purpose of these descriptors is to encode or categorize with lexemes and syntagms the iconic visual content of artworks. Therefore, artwork iconic analysis is achieved through the analysis of these semantic annotations. A recent, innovative initiative to this approach was the creation by Hébert and Trudel of a new digitized corpus of the Belgian painter René Magritte’s artworks (Hébert 2013; Hébert and Trudel 2013). This corpus consists of 1,780 artworks5, almost all the artworks of the painter, mainly oils, gouaches, drawings, sketches and posters, whose date of creation varies from 1916 to 1967. The corpus is based on David Sylvester’s catalogue raisonné (Sylvestre 1997). Table 2 illustrates some examples of Magritte’s artworks and their semantic indexing produced by Hébert and Trudel. Each piece of art is associated with a title, an identifier, a date and a semantic annotation. Each annotation is composed of several descriptors delimited by "< >" tags. The semantic indexing was done in French, but we translated the descriptors to the best of our knowledge. The original titles of Magritte’s artworks have been left unchanged even when they were in French.

4

It is the Large Scale Visual Recognition Challenge (www.image-net.org/challenges/LSVRC/). This is a prestigious competition about a semantic indexing task of images from the ImageNet dataset. 5 In the database built by Hébert and his team, 1,957 artworks are listed, but only 1,780 have been indexed so far. 12

Table 2: Three examples of Magritte’s artworks and their semantic indexing. Id

383

27

304

Artwork title

Golconde

Baigneuses

La trahison des images

Date

Image

Semantic annotation

1953

, , , , , , , , , , , , , , , , , , , , , , ,

1921

, , , , , , , , , , , , , , , , , , , , , , , ,

1929

, , , , , , , , , , , , , , , , ). Table 4: The top 15 nearest Magritte’s artworks in the SVS to the syntagmatic signature vector of , that are not indexed with the descriptor . #137

#1170

#1648

#1546

#1920

sens de la nuit, Le

idée, L'

terre d'Osiris, La

maître d'école, Le

[Publicity design for Norine: “Lord Lister”]

cos=0.524

cos=0.442

cos=0.401

cos=0.390

cos=0.367

#1178

#1137

#880

#189

#709

art de vivre, L'

pan de nuit, Le

philtre, Le

barbare, Le

ellipse, L'

cos=0.359

cos=0.350

cos=0.346

cos=0.345

cos=0.343

#1158

#114

#78

#1184

#141

bouchon d'épouvante, Le

conquérant, Le

grande nouvelle, La

[Man seated at a table]

fatigue de vivre, La

cos=0.343

cos=0.339

cos=0.317

cos=0.315

cos=0.307

18

Recall that the descriptor has a very specific syntagmatic signature, its most (statistically) important co-occurrences are , , , and so on. A cosine-based paradigmatic analysis allows us to predict that we should observe a man in an artwork or something permutable to it whenever its indexation is characterized by that particular syntagmatic signature or combinatorial pattern. 4.4. Discussion The results illustrated in Table 4, although limited to the silences of a single descriptor, are very revealing, to the point that we may wonder whether Hébert and Trudel have not committed significant indexing errors. For example, why are the artworks "Le sens de la nuit" (# 137) and "Le maître d’école" (# 1546) not indexed with the descriptor , despite the fact that we find in these artworks all the features of its syntagmatic signature? These artworks clearly contain the iconic sign of a man in Magritte’s artworks and with an obvious figurativeness7. Hence, these artworks should have been indexed with the descriptor . According to our method, these absences are silences in Hébert and Trudel’s indexing. Furthermore, other artworks listed in Table 4 are not necessarily characterized by silences. These artworks rather illustrate another kind of relation with . As we can see in many artworks of Table 4, for instance # 1170, # 1648, # 1178, # 1137 and # 880, there is a semiotic phenomenon that we had not expected to find. These artworks do not contain any iconic sign designating with obvious figurativeness the presence of a man. We are rather confronted with an elliptic phenomenon evoked by the full or partial presence of the syntagmatic signature of the man (important features of its combinatorial pattern) despite the absence of the signifier. What this cosine-based paradigmatic analysis allows us to discover here is not indexing errors, but rather a complex rhetorical phenomenon that opens up an ontological distinction of the iconic visual sign: the figurative iconic sign and the abstract iconic sign (with low figurativeness). This distinction recalls the opposition between iconization and abstraction of visual signs (Greimas, Collins, and Perron 1989:634). This is a very interesting unexpected result. It provides new insights for future works about a computational semiotics such as the one we are trying to develop in this paper.

7

We used the concept of figurativeness in the greimassian way : signs that « permit us to interpret it as representing an object of the natural world ». (Greimas, Collins, and Perron 1989, 634). 19

5. Experimentation # 2: The Componential Analysis The second experiment studies another kind of semiotic phenomenon. The objective of this experiment is to show how the previously defined orthogonal complement subtraction operator can be used by an algorithm to produce a componential analysis of the descriptors used to index Magritte’s artwork iconic signs. The results of the experiment suggest that this subtraction operator enables the syntagmatic signature decomposition into its various semantic components. 5.1. Problem Introduction Some descriptors have a complex semantic, they seem to index different iconic signs in Magritte’s artworks. Let us take a look at the illustration the descriptor in the studied corpus. The syntagmatic signature of this descriptor is projected in a very specific region of the SVS. Table 5 illustrates its nearest neighbors in the SVS as calculated with the cosine. Table 5: The top 15 nearest descriptors of in the semantic vector space of the annotations of Magritte’s artworks. Descriptor

𝐜𝐨𝐬(< woman >, 𝐯j )



1.000



0.824



0.795



0.783



0.765



0.762



0.759



0.759



0.732



0.707



0.652



0.590



0.588

20



0.576



0.562

The descriptors that populate the region where is projected suggest that its syntagmatic signature is very similar to the combinatorial patterns of other descriptors such as , , , , and . These results will not surprise many experts of Magritte’s artworks either. While man usually comes in a threepiece suit (as seen in the previous experimentation), the woman is usually naked in Magritte’s artworks. However, if we look closely at all of the nearest neighbors of sorted in Table 5, we see other descriptors such as and that express something different than the feminine nudity. The syntagmatic signature of appears to be projected at the edge of two regions in the SVS. In one of these regions, we find descriptors that index feminine nudity, but in the second one, more peripheral, we find descriptors that index the woman face. This suggests a new research question: could the syntagmatic signature of a descriptor be componential? In other words, could the syntagmatic signature of a sign hide several components under a dominant combinatorial pattern? The aim of the second experiment is to develop a method of componential analysis that decomposes the syntagmatic signature of a descriptor into its different combinatorial subpatterns called its semantic component. 5.2. Method By definition, any vector can be decomposed into a set of orthogonal vector components. Formally, this is what our method wants to accomplish, but applied to the decomposition of the syntagmatic signature vector of a target descriptor 𝐯i into a subset of "vector components" 𝐮1 + 𝐮2 + ⋯ + 𝐮m corresponding to the syntagmatic vectors of other descriptors present in Hébert and Trudel’s annotations of Magritte’s artworks. For example, could the syntagmatic signature of be decomposed into the following vector components: = ( + )? This is the kind of question our method is designed to answer. To do so, let us denote the composition of the syntagmatic signature of a descriptor in the following way: 𝐂𝐎𝐌𝐏𝐎𝐒𝐈𝐓𝐈𝐎𝐍(𝐯i ) ≈ (𝐮1 + 𝐮2 + ⋯ + 𝐮m ), where 𝐯i is the syntagmatic signature vector of the target descriptor i we want to decompose and 𝐮1 + 𝐮2 + ⋯ + 𝐮m are vector components corresponding to sub-combinatorial patterns of the target. To find 𝐂𝐎𝐌𝐏𝐎𝐒𝐈𝐓𝐈𝐎𝐍(𝐯i ) we use the negation operator introduced above which we applied recursively using the Gram-Schmidt algorithm until the decomposition process does not find any new component. The method proceeds as follows. In a first iteration, it searches with the cosine operator 𝐜𝐨𝐬(𝐯i , 𝐮j ) the descriptor 𝐮j with the syntagmatic signature vector the most similar to the target 𝐯i we want to decompose. The vector 𝐮j then becomes the first component of 𝐯i . We decompose 𝐯i using the negation operator 𝐃𝐈𝐅𝐅(𝐯i , 𝐮j ) = 𝛆𝒕 , where 𝛆𝒕 represents the residual 21

of 𝐯i , that is to say the syntagmatic signature of the target 𝐯i orthogonal to 𝐮j . In a second iteration, the method searches a new descriptor whose syntagmatic signature vector 𝐮k is the most similar to the residual 𝛆𝒕 obtained at the previous iteration. The vector 𝐮k becomes the second component of 𝐯i . The negation operator is again applied, but this time on the residual 𝐃𝐈𝐅𝐅(𝛆𝒕 , 𝐮k ) = 𝛆𝒕+𝟏 , which gives a new residual corresponding to the syntagmatic signature of 𝐯i orthogonal to 𝐮j and to 𝐮k . As noted by Widdows, in order to respect the commutativity of the negation operator at each iteration, it is important to apply the gram-Schmidt’s orthogonalization algorithm on every new vector component retrieved (Widdows 2004:230). We repeat the iteration until 𝐜𝐨𝐬(𝛆𝒕 , 𝛆𝒕+𝟏 ) ≈ 1.0, that is, until the decomposition process stabilizes. This occurs when the residual 𝛆𝒕+𝟏 can no longer be decomposed using the syntagmatic signature vector of another descriptor of the corpus. At the end of the process, the composition of the syntagmatic signature of the target descriptor corresponds to the following vector addition (𝐯i ) ≈ (𝐮𝑗 + 𝐮𝑘 + ⋯ + 𝛆𝒕+𝟏 ). 5.3. Results We performed this analysis on the descriptor . As mentioned earlier, according to the cosine operator, the most similar descriptor to is with cos(, )=0.82. This cosine indicates that the two descriptors are not perfectly equivalent (i.e. permutable) in the corpus, hence the syntagmatic signature vector of can not be reduced to that of although the latter is an important component of the former. By retrieving in a second iteration the negation 𝐃𝐈𝐅𝐅(< woman >, < nudity >), we obtain a new syntagmatic signature vector for which is very different from that of . In the region of the SVS where the vector 𝐃𝐈𝐅𝐅(< woman, < nudity >) is, we find descriptors such as , , , and that have nothing to do with nudity (see Table 6). These descriptors all express another semantic component of , that is, as we suspected, related to the expression of the woman face. Note, however, that it is not the descriptor that is selected by our method, but a highly correlated descriptor to . The most intuitive way to illustrate the results of our method is by retrieving artworks whose annotations are located in the region of , of 𝐃𝐈𝐅𝐅(< woman >, < nudity >) and of 𝐃𝐈𝐅𝐅(< woman >, < hair >).These artworks are sorted in Table 7: Table 6: Top 10 nearest descriptors in the semantic vector space of the negation of by and the double negation of by and 𝐃𝐈𝐅𝐅(< woman >, < nudity >) = 𝛆𝒕

𝐃𝐈𝐅𝐅(𝐃𝐈𝐅𝐅(< woman >, < nudity >), < hair >) = 𝛆𝒕+𝟏

Descriptors

𝐜𝐨𝐬(𝐯j , 𝛆𝒕 )

Descriptors

𝐜𝐨𝐬(𝐯j , 𝛆𝒕+𝟏 )



0.566



0.486



0.449



0.207



0.440



0.203 22



0.403



0.192



0.371



0.173



0.343



0.165



0.335



0.165



0.335



0.152



0.333



0.147



0.321



0.145

Table 7: Top 10 nearest artworks in the semantic vector space of the two main semantic components of . 𝐜𝐨𝐬(< woman >, 𝐨𝐢 )

𝐜𝐨𝐬(𝐃𝐈𝐅𝐅(< woman >, 𝐜𝐨𝐬(𝐃𝐈𝐅𝐅(< woman >, < hair >), 𝐨𝐢 ) < nudity >), 𝐨𝐢 )

#738

#22

#768

menottes de cuivre, Les

[Portrait of Emma Bouillon]

peinture, La

cos=0.880

cos=0.490

cos=0.707

#743

#635

#497

menottes de cuivre, Les

[Portrait of Jacqueline Nonkels]

représentation, La

23

cos=0.877

cos=0.483

cos=0.695

#742

#634

#1862

menottes de cuivre, Les

[Portrait of Georgette Magritte]

statue volante, La

cos=0.875

cos=0.481

cos=0.659

#303

#760

#1205

femme cachée, La

‹mémoire, La›

folie des grandeurs, La

cos=0.875

cos=0.475

cos=0.658

#756

#1198

#91 24

Femme-bouteille

[Head]

‹Souvenir de voyage›

cos=0.872

cos=0.467

cos=0.657

#758

#1197

#408

Femme-bouteille

[Head]

belle de nuit, La

cos=0.872

cos=0.463

cos=0.651

#755

#767

#1057

Femme-bouteille

mémoire, La

folie des grandeurs, La

cos=0.868

cos=0.461

cos=0.650

25

#418

#637

#197

viol, Le

[Portrait of Eliane Peeters]

éloge de l'espace, L'

cos=0.866

cos=0.461

cos=0.650

#770

#61

#409

Femme-bouteille

Portrait d'Evelyne Bourland

Quand l'heure sonnera

cos=0.865

cos=0.460

cos=0.647

#761

#4

#798

26

Femme-bouteille

[Plaster bust and fruit]

leçon des choses, La

cos=0.864

cos=0.458

cos=0.647

5.4. Discussion The artworks in Table 7 constitute a very convincing demonstration of the ability of the orthogonal complement subtraction operator to infer the semantic composition of a descriptor. The first column in Table 7 lists the top 10 nearest artworks to in the SVS, the middle column, the top 10 nearest artworks to the negation by and the third column, the top 10 nearest artworks to the negation of by . Artworks in the first column are a mixture of the two semantic components of the syntagmatic signature of . In the second column, we find not only artworks expressing faces of women, but also artworks without any feminine nudity, whereas the opposite is true in the artworks of the third column where women are naked but faceless. Does the syntagmatic signature vector of have a third component? Our method indicates that it does not, that is 𝐜𝐨𝐬(𝛆𝒕 , 𝛆𝒕+𝟏 ) ≈ 1.0. This means that the region of the SVS where the residual of the double negation 𝐃𝐈𝐅𝐅(𝐃𝐈𝐅𝐅(< woman >, < nudity >), < hair >) clusters descriptors whose syntagmatic signatures differ very little from that of or (see Table 6). The residual seems to represent an atypical variant of the preceding components rather than a new component orthogonal to the preceding ones. There is no artwork in the studied corpus in which the woman is represented without a syntagmatic signature correlated to or correlated to . The results of the semantic decomposition of , although limited as in the first experiment to the analysis of a single descriptor, are very convincing. They enable us to reach our second objective, which was to show how an algorithm based on the orthogonal complement subtraction operator can produce a componential analysis of the descriptors used to index the iconic content of Magritte’s artworks. However, the definition given to the componential relation has its limits. On the one hand, it is very unlikely that in Hébert and Trudel’s corpus the syntagmatic signature vector of a descriptor 𝐯i is perfectly orthogonal to a subset of other descriptor vectors 𝐮1 + 𝐮2 + ⋯ + 27

𝐮m , and, on the other hand, it is also highly unlikely that these components are orthogonal to one another. This is the case of for instance, which is not perfectly decomposable by the and components. There is not a perfect equivalence between the two since 𝐜𝐨𝐬(< femme >, (< nudité > +< cheveux >)) = 0.86 (an equivalence would assume a cosine of 1.0) and there is no orthogonality between its components since 𝐜𝐨𝐬(< nudité >, < cheveux >) = 0.48 (and the orthogonality would assume a cosine of 0.0). The components retrieved by our method are quasi-orthogonal. Nevertheless, conjecturing that the syntagmatic signature vector of "woman" has two main semantic components, the combinatorial pattern of and the combinatorial pattern of , represents according to our method a very likely hypothesis about the componential structure of . It invites then specialists of Magritte’s artworks to consider it as an informed conjecture. 6. Experimentation # 3: The Topic Analysis The semiotic phenomenon studied in this last experiment is topic modelling. The aim of this experiment is to show how a clustering algorithm based on comparison and composition operators can produce a topic analysis of Magritte’s artworks. The results obtained show that the rationale explaining why a clustering algorithm can produce this kind of semiotic analysis is the isotopic process that supports topic modelling and that a clustering algorithm seems to reconstruct. 6.1. Problem Introduction Descriptors and artwork annotations from Hébert and Trudel’s semantic indexing are not uniformly distributed in the SVS. There are groups of descriptors and groups of artwork annotations that, because they share similar combinatorial patterns in the corpus, they are projected into the same regions of the SVS. Therefore, the SVS is structured by various highdensity regions. Figure 4 illustrates this phenomenon partially. Small triangles represent artwork annotations of Magritte and circles descriptors. The coordinates of these triangles and circles encode their vector representations. Although it is a two-dimensional reduction of the total SVS, we can nevertheless see some high-density regions where artwork annotations related to a common topic are clustered together. For example, we see at the left end a region where artwork annotations are clustered around a small number of descriptors such as , , , , , , and . We also see in the lower right corner another region where artwork annotations are grouped around descriptors such as , , , , , , and . The first region includes artworks that share a specific topic one could summarize with the predicate "FEMALE NUDITY", while the second region groups artworks that share another specific topic one could summarize with the predicate "RIDING HORSE".

28

FEMALE NUDITY

RIDDING HORSE

Figure 4: Multidimensional scaling of the semantic vector space of the annotations of Magritte’s artworks.

What are the other regions of the SVS of Magritte’s artworks annotated by Hébert and Trudel? Do these regions also cluster together artworks that share a common topic? The visualization of Figure 4 does not allow us to go much further in this analysis, because the two-dimensional reduction is too distorted. To discover other high-density regions in the SVS, we must use a clustering algorithm. 6.2. Method Let us denote a space partition into k regions by ℙ = {𝑅1 , … 𝑅𝑘 }, the vector representation of an artwork by the descriptor vectors addition 𝐨j = ∑𝑣i∈𝑐𝑗(𝐯i ) and let us define a region 𝑅𝑖 as a convex surface that satisfies the following condition: 𝑅𝑖 = {𝐨 ∈ 𝑅𝑖 : ∀ 𝐜𝐨𝐬(𝐨, 𝐜i ) > 𝐜𝐨𝐬(𝐨, 𝐜j )} 𝑅𝑗 ∈ℙ 𝑗≠i

𝐜i =

1 ∑𝐨 |𝑅𝑖 | 𝐨∈𝑅𝑖

Where the vector 𝐜j represents the geometric center of the region 𝑅𝑖 (its centroid). A vector space partition into convex regions produces a Voronoi structure which has very interesting properties for a topic modelling analysis. Firstly, the convexity properties of this partition guarantee that the geometric center of a region is its most representative vector. Consequently, it is a very natural way to represent the topic associated to a region: a topic is represented by the vector averaging a group of very nearby artworks in the SVS, a group of artworks that share a very similar syntagmatic signature vector. Secondly, because of the convexity property of these regions, the cosine operator allows what Gärdenfors called “prototype inferences” (Gärdenfors 2000): in convex regions, the distance between the syntagmatic signature vector of a descriptor and the center of a region is correlated to its semantic representativeness. In other words, the closer a descriptor is to the center of a region, the more it is representative of the topic common to the artworks clustered in the concerned region of the SVS.

29

The K-means clustering algorithm generates exactly this kind of structure. Its objective function is the following: argmax ∑ ∑ 𝐜𝐨𝐬(𝐨i , 𝐜j ) 𝑅𝑗 ∈ℙ 𝐨i ∈𝑅𝑗

The algorithm is a mobile centers iterative procedure which seeks to move the geometrical centers 𝐜j of each region 𝑅𝑗 in a partition 𝑃 in order to iteratively maximizes 𝐜𝐨𝐬(𝐨i , 𝐜j ). The objective function of K-means guarantees, for a partition into k regions and a given seed, an optimal partition.8 6.3. Results We applied the K-means algorithm to the artwork annotations produced by Hébert and Trudel and obtained a partition of the SVS into 20 regions.9 According to our method, this indicates that Magritte’s artworks are organized around 20 main topics. Each topic is represented in Table 8 as a list of its 10 most representative descriptors, that is the 10 nearest descriptors to the center of each region of the SVS (as calculated with the cosine). We also added a predicate in capital letters that summarizes the meaning structure of each topic. These topics are: (0) MASONRY, (1) BOISERIE, (2) FACE, (3) LEAF, (4) EYE, (5) RIDING HORSE, (6) SEA, (7) PIPE, (8) FIRE, (9) TREE, (10) SHEET MUSIC, (11) LION, (12) RIPPLE, (13) FEMALE NUDITY, (14) DRESSED MAN, (15) HOUSE, (16) PIANO, (17) WORD INSCRIPTIONS, (18) SKY, and (19) GEOMETRICAL SHAPE. Finally, we also retrieved the five most representative artworks for each topic, that is the five nearest artworks in the SVS to the center of each region. These results are available in Appendix A. Table 8: Topics in the annotations of Magritte’s artworks and the top 10 nearest descriptors of each region center. Topic #0

8 9

Topic #1

Topic #2

Topic #3

MASONRY

Cos BOISERIE

cos FACE

cos LEAF

cos



0.51



0.48

0.71

0.51

0.50



0.48

0.69

0.51





0.47

0.67

0.50

0.46



0.45

0.65

0.49



0.43

0.63

0.48

0.49

0.40

Centroid seeding was estimated using the algorithm of (Arthur and Vassilvitskii 2007). The estimation the k parameter was obtained by maximizing the Silhouette coefficient (Rousseeuw 1987). 30



0.39



0.40



0.36



0.39 0.59

0.41



0.32



0.39

0.59

0.41



0.31



0.37

0.58

0.41



0.31

0.34

0.51

0.41

Topic #4

Topic #5

0.63 0.46

Topic #6

Topic #7

EYE

cos RIDDING HORSE cos SEA

cos PIPE



0.61

0.59

0.62 0.81



0.60

0.56

0.60 0.81



0.60

0.54

0.59

0.81



0.53

0.54

0.55

0.81



0.51

0.54

0.47

0.81



0.51

0.51

0.47

0.80



0.49

0.51

0.44

0.77



0.47

0.49

0.40

0.67



0.47


Topic #9

Topic #10

cos

Topic #11

FIRE

cos TREE

cos SHEET MUSIC cos LION

cos



0.39

0.57

0.64

0.50



0.39

0.56

0.64

0.44



0.33

0.50

0.64 0.43



0.33

0.48

0.64

0.42

0.40

0.59

0.41



31

0.28

0.36

0.52

0.40



0.27

0.36

0.52

0.40



0.27

0.36

0.50

0.40



0.26

0.35

0.48

0.40



0.25

0.35

0.37

0.40

Topic #12

Topic #13

Topic #14

Topic #15

RIPPLE

cos FEMALE NUDITY cos DRESSED MAN cos HOUSE

cos



0.34

0.77

0.43

0.59



0.33

0.76

0.41

0.58



0.32

0.75

0.40

0.57



0.31

0.74

0.40

0.55



0.27

0.74

0.37

0.54



0.26

0.73

0.36

0.49

0.26

0.73

0.35 0.48



0.25

0.73

0.35

0.44



0.25

0.71

0.33

0.44



0.25

0.65

0.32

0.44

Topic #17 Topic #16

Topic #19 Topic #18

PIANO

cos

WORD INSCRIPTION

cos SKY

cos

GEOMETRICAL SHAPE

cos



0.94



0.33

0.57



0.72



0.94



0.19

0.49



0.72



0.94



0.19

0.37



0.69



0.94



0.18

0.36



0.68



0.94



0.18

0.26



0.46

32



0.93



0.18

0.24



0.43



0.93



0.18

0.23



0.38



0.93



0.17

0.23



0.37

< keyboard_(piano)>

0.91



0.16

0.21



0.26



0.89



0.16

0.20



0.26

6.4. Discussion The results in Table 8 and the artworks listed in Appendix A suggest that a clustering algorithm like K-means applied to the SVS makes it possible to infer the main topic structures that organize Magritte’s artworks. All lists of descriptors in Table 8 and artworks in Appendix A are characterized by remarkable topic coherence. Many have noticed this correlation phenomenon between regions of a SVS and topic structures, but only in the context of textual corpus analysis (Burgess et al. 1998; Griffiths et al. 2007; Larsen and Monarchi 2004; Rieger 1983; Widdows 2004:4). Some researchers have conjectured that the explanation of this correlation is the isotopic process that supports topic modelling (Mayaffre 2008; Pincemin 1999). The rationale why a clustering algorithm can produce such semiotic analysis is because of its partial reconstruction of the isotopic process. Isotopy is the process of repetition in several contexts of similar semantic feature combinations (known as “semes”) (Rastier 1996). A topic is the meaning structure supported by this process. The nature of the relations between these semantic features can be very complex and this complexity is not encoded in an SVS model because all relations are reduced to co-occurrence. Nevertheless, what the previous results demonstrate is that an algorithm as K-means can approximate in a very convincing way this isotopic process. An algorithm like K-means clusters together artwork annotations (the contexts) characterized by similar patterns of co-occurrence of descriptors. Each centroid of artwork cluster is a vector structure - a geometrical center - generated by this clustering process. This is the reason why we consider these centroids as the vector representation of the topics in Magritte’s artworks. For example, the region corresponding to the topic #4 clustered together several artworks where a repeated combinatorial pattern with some variations is present. This pattern consists in variants of descriptor combinations of , , and so on. The meaning structure supported by these combinatorial patterns is not encoded by the vector of any particular descriptor in the corpus, but we can easily recognize it and index it with a predicate. We have used the predicate EYE for this purpose. These results are interesting for specialists of Magritte’s artworks and allow us to formulate several research questions. For example, which topics in Magritte’s artworks are the most important? In other words, which regions of the SVS contains the largest number of 33

Magritte’s artworks? The graph below (Figure 5) shows that the most important topic in Magritte’s artworks is FEMALE NUDITY. This is the main topic of approximately 13% of Magritte’s artworks. The second most important topic is the FACE topic, which is the main topic of 11% of Magritte’s artworks. Together they represent the main topics of a quarter of Magritte’s artworks. The topics LEAF, SEA, TREE, HOUSE, MASONRY, SKY, DRESSED MAN and BOISERIE represent each between 6% to 8% of Magritte’s artworks. The rest are relatively rare and represent the main topic of 1% to 4% of Magritte’s artworks. Unfortunately, in this article, we cannot go much deeper in the analysis of these results. The primary objective was methodological. It consisted in demonstrating that one can produce a topic analysis with a clustering algorithm applied on the SVS.

14%

120%

12%

100%

10%

80%

8%

60% 6% 40%

4% 2%

20%

0%

0%

Cumulative Proportion

Topic Proportion

The proportion of each topic in Magritte's artworks

Topics

Figure 5: The proportion of each topic in Magritte’s artworks.

7. Conclusion The objective of this article was to explore the contribution of the SVS model and its algorithms to a data-driven computational semiotics. We wanted to explore what kind of semiotic analysis one can produce with this framework. In order to provide some answers to this question, we first presented the SVS theoretical model, its main parameters and the main operators of its semantic inference engine. The SVS framework has generally been presented in a perspective limited to linguistics or information retrieval and in the context of textual corpus analysis. We have tried to detach the SVS from this particular context. To achieve this, we have applied the SVS and its algorithms to an new corpus composed of the surrealist painter René Magritte’s artworks. To our knowledge, this is the first time the SVS has been applied to the analysis of iconic signs in artworks. This is an important contribution because it shows that the SVS constitutes a framework that can be generalized to the analysis of other types of signs that are not strictly language-based. 34

We conducted three short experiments on this corpus. Each was designed to explore a type of semiotic analysis on the corpus of Magritte’s artworks with algorithms based on the basic algebraic operators of the SVS framework. The first experiment was a cosine-based paradigmatic analysis. It showed how one can discover silences in Hébert and Trudel’s indexing of Magritte’s artworks. The second experiment was a componential analysis. It was based on a semantic decomposition algorithm. This experiment showed how the componential structure of the syntagmatic signature vector of a descriptor could be inferred with the orthogonal complement subtraction operator. Finally, the third experiment was a topic modelling analysis. It was based on a clustering algorithm. This experiment showed that the regions of the SVS induced with a clustering algorithm allows us to discover the topic structures organizing Magritte’s artworks. These experiments only represent a fraction of the semiotic analyses one can produce within this framework. For example, the results of the first experiment suggested that we could also use the SVS framework to analyze elliptic structures. Other researches have also suggested that the SVS framework can allow the analysis of metaphor (Shutova 2010), analogy (Mikolov et al. 2013), hypernymy (Santus, Lenci, et al. 2014), hyponymy (Erk 2009) and more. This is very promising for the development of a data-driven computational semiotics. The results are both theoretical and methodological. Theoretically, the SVS framework constitutes a very powerful mathematical model for semiotics: the linear algebra. Methodologically, the algorithms of the SVS framework allow a data-driven approach within which large-scale semiotic analysis is possible. Encouraged by the results of our experiments, we believe that future works should pursue the investigation of other kinds of semiotic analysis one can produce with the SVS framework. Appendix Appendix A: Five Nearest Artworks of Each Topic in the Semantic Vector Space. 0-masonry id

1291

848

1653

944

1479

title témoin, Le

confort de l'esprit, Le

sac à malice, Le

leçon des ténèbres, entrée en scène, L' La

cos 0.630

0.621

0.619

0.619

0.612

35

1-boiserie id

430

1751

1739

298

1298

title

palais de rideaux, Le

palais de rideaux, Le

vengeance, La

[Interior with standing object]

vengeance, La

0.624

0.622

0.622

0.621

cos 0.638

2-face id

219

254

1141

1595

637

title

paysage fantôme, Le

amants, Les

porte ouverte, La

Georgette

[Portrait of Eliane Peeters]

0.782

0.775

0.773

0.773

580

1415

736

1418

cos 0.784

3-leaf id

1323

36

title île au trésor, L'

île au trésor, L'

‹île au trésor, L'›

grâces naturelles, Les

prince charmant, Le

cos 0.690

0.689

0.686

0.684

0.682

1380

1387

1382

750

title Shéhérazade

Shéhérazade

Shéhérazade

Shéhérazade

Objet peint : œil

cos 0.732

0.719

0.718

0.710

0.708

1771

1134

647

1039

title jockey perdu, Le

blanc-seing, Le

blanc-seing, Le

tour d'ivoire, La

enfance d'Icare, L'

cos 0.699

0.686

0.668

0.654

0.650

4-eye id

1385

5-ridding horse id

567

37

6-sea id

1001

1548

938

1632

989

title idées claires, Les

rappel à l'ordre, Le

origines du langage, Les

origine du langage, monde familier, L' Le

cos 0.730

0.729

0.723

0.721

0.719

7-pipe id

304

1492

1639

1555

1954

title

trahison des images, La

trahison des images, La

trahison des images, La

‹trahison des images, La›

[Poster for a shop window]

0.839

0.835

0.835

0.834

235

119

1600

449

title belle captive, La

paysage en feu, Le

aube à Cayenne, L'

aube à Cayenne, L'

condition humaine, La

cos 0.479

0.464

0.453

0.453

0.451

cos 0.839

8-fire id

1460

38

9-tree id

1732

543

1680

1745

617

title

chœur des sphinges, Le

grandes espérances, Les

‹clairvoyance, La›

recherche de l'absolu, La

vie heureuse, La

0.740

0.728

0.724

0.723

cos 0.754

10-sheet music id

1827

1787

1826

1808

1810

title

premiers amours, Les

Sans titre

reconnaissance infinie, La

misanthropes, Les

trio magique, Le

0.707

0.706

0.699

0.695

488

1274

1296

1275

cos 0.711

11-lion id

492

39

jeunesse illustrée, La

jeunesse illustrée, La

messagère, La

jeunesse illustrée, La

0.612

0.595

0.594

0.589

92

1211

177

1105

Joconde, La

déesse des environs, La

soir qui tombe, Le

0.545

0.541

0.540

0.537

644

653

652

435

title aimant, L’

viol, Le

rêve, Le

aimant, L'

viol, Le

cos 0.849

0.846

0.846

0.845

0.844

title

jeunesse illustrée, La

cos 0.621

12-ripple id

109

title

catapulte du désert, À la suite de l'eau, La les nuages

cos 0.572

13-female nudity id

556

40

14-dressed man id

137

1251

868

876

1829

title sens de la nuit, Le

[Design for a mural]

chant de la violette, Le

Journal intime

[Title unknown]

cos 0.618

0.606

0.604

0.597

0.596

15-house id

838

974

958

1554

1702

title

empire des lumières, L'

empire des lumières, L'

empire des lumières, L'

parabole, La

ombre monumentale, L'

0.765

0.763

0.757

0.756

1506

1522

910

1936

cos 0.778

16-piano id

1547

41

title main heureuse, La

main heureuse, La

main heureuse, La main heureuse, La

[Pleyel]

cos 0.963

0.962

0.962

0.961

0.868

211

436

1789

256

Sans titre

monde perdu, Le

17-word inscription id

318

title

arbre de la science, bonne nouvelle, La miroir vivant, Le L'

cos 0.400

0.377

0.368

0.367

0.366

1136

156

511

203

18-sky id

149

title Campagne

Parmi les bosquets palais de rideaux, légers Le

chant de l'orage, Le

paysage secret, Le

cos 0.642

0.622

0.616

0.611

0.622

42

19-geometrical shape Id

1239

83

16

1243

53

title

[Flowers in front of a window]

[Landscape]

forgerons, Les

Retraite militaire

[Abstract composition]

0.758

0.758

0.758

0.750

cos 0.758

References Arora, Ravneet Singh. 2012. “Towards Automated Classification of Fine-Art Painting Style: A Comparative Study.” Rutgers University-Graduate School-New Brunswick. Bar, Yaniv, Noga Levy, and Lior Wolf. 2014. “Classification of Artistic Styles Using Binarized Features Derived from a Deep Neural Network.” Pp. 71–84 in Workshop at the European Conference on Computer Vision. Springer. Baroni, Marco and Alessandro Lenci. 2010. “Distributional Memory: A General Framework for Corpus-Based Semantics.” Computational Linguistics 36(4):673–721. Blei, David M. and John D. Lafferty. 2009. “Topic Models.” Text Mining: Classification, Clustering, and Applications 10(71):34. Bordag, Stefan and Gerhard Heyer. 2007. “A Structuralist Framework for Quantitative Linguistics.” Pp. 171–189 in Aspects of Automatic Text Analysis. Springer. Bouma, Gerlof. 2009. “Normalized (Pointwise) Mutual Information in Collocation Extraction.” Proceedings of GSCL 31–40. Burgess, Curt. 2000. “Theory and Operational Definitions in Computational Memory Models: A Response to Glenberg and Robertson.” Journal of Memory and Language 43(3):402–8. Burgess, Curt, Kay Livesay, and Kevin Lund. 1998. “Explorations in Context Space: Words, Sentences, Discourse.” Discourse Processes 25(2–3):211–57. Carneiro, Gustavo, Nuno Pinho da Silva, Alessio Del Bue, and João Paulo Costeira. 2012. “Artistic Image Classification: An Analysis on the Printart Database.” Pp. 143–57 in Computer Vision–ECCV 2012. Springer. 43

Crowley, Elliot J. and Andrew Zisserman. 2014. “The State of the Art: Object Retrieval in Paintings Using Discriminative Regions.” in British Machine Vision Conference. De Souza, Clarisse Sieckenius. 2005. The Semiotic Engineering of Human-Computer Interaction. MIT press. Dunning, Ted. 1993. “Accurate Methods for the Statistics of Surprise and Coincidence.” Journal Computational Linguistics 19(1):61–74. Erk, Katrin. 2009. “Supporting Inferences in Semantic Space: Representing Words as Regions.” Pp. 104–15 in Proceedings of the Eighth International Conference on Computational Semantics. Association for Computational Linguistics. Evans, James A. and Pedro Aceves. 2016. “Machine Translation: Mining Text for Social Theory.” Annual Review of Sociology 42:21–50. Firth, J. R. 1957. “A Synopsis of Linguistic Theory, 1930-1955.” Studies in Linguistic Analysis Special Volume:1–32. Floridi, Luciano. 1999. Philosophy and Computing: An Introduction. Psychology Press. Gärdenfors, P. 2000. Conceptual Spaces: The Geometry of Thought. Cambridge: MIT Press. Gärdenfors, P. 2014. The Geometry of Meaning: Semantics Based on Conceptual Spaces. MIT Press. Graham, Daniel J., Jay D. Friedenberg, Daniel N. Rockmore, and David J. Field. 2010. “Mapping the Similarity Space of Paintings: Image Statistics and Visual Perception.” Visual Cognition 18(4):559–73. Greimas, Algirdas Julien, Frank Collins, and Paul Perron. 1989. “Figurative Semiotics and the Semiotics of the Plastic Arts.” New Literary History 20(3):627–649. Griffiths, Thomas L., Mark Steyvers, and Joshua B. Tenenbaum. 2007. “Topics in Semantic Representation.” Psychological Review 114(2):211. Groupe Mu. 1992. Traité du signe visuel : Pour une rhétorique de l’image. Paris: Seuil. Hamilton, William L., Jure Leskovec, and Dan Jurafsky. 2016. “Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change.” in Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, vol. 1489–1501. Berlin, Germany. Hare, Jonathon S., Paul H. Lewis, Peter GB Enser, and Christine J. Sandom. 2006. “Mind the Gap: Another Look at the Problem of the Semantic Gap in Image Retrieval.” Pp. 607309-607309–12 in Electronic Imaging 2006. International Society for Optics and Photonics. Harris, Zellig S. 1951. Methods in Structural Linguistics. Chicago & London: The University of Chicago Press. Harris, Zellig S. 1954. “Distributional Structure.” Word 10(23):146–62.

44

He, Kaiming, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. “Deep Residual Learning for Image Recognition.” Pp. 770–778 in Proceedings of the IEEE conference on computer vision and pattern recognition. Hébert, L. 2013. “Magritte. Toutes Les Œuvres, Tous Les Thèmes.” Retrieved March 16, 2016 (www.magrittedb.com). Hébert, L. and Éric Trudel. 2013. “Analyse Des Images”. Magritte. Toutes Les Œuvres, Tous Les Thèmes. Rimouski (Québec): Université du Québec à Rimouski. Retrieved March 15, 2016 (http://magrittedb.com). Johnson Jr, C.Richard et al. 2008. “Image Processing for Artist Identification.” Signal Processing Magazine, IEEE 25(4):37–48. Kell, Douglas B. and Stephen G. Oliver. 2004. “Here Is the Evidence, Now What Is the Hypothesis? The Complementary Roles of Inductive and Hypothesis‐driven Science in the Post‐genomic Era.” Bioessays 26(1):99–105. Kelling, Steve et al. 2009. “Data-Intensive Science: A New Paradigm for Biodiversity Studies.” BioScience 59(7):613–20. Khan, Fahad Shahbaz, Shida Beigpour, Joost van de Weijer, and Michael Felsberg. 2014. “Painting-91: A Large Scale Database for Computational Painting Categorization.” Machine Vision and Applications 25(6):1385–97. Kiela, Douwe and Stephen Clark. 2014. “A Systematic Study of Semantic Vector Space Model Parameters.” Pp. 21–30 in Proceedings of the 2nd Workshop on Continuous Vector Space Models and their Compositionality (CVSC) at EACL. Kitchin, Rob. 2014. “Big Data, New Epistemologies and Paradigm Shifts.” Big Data & Society 1(1). Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. 2012. “Imagenet Classification with Deep Convolutional Neural Networks.” Pp. 1097–1105 in Advances in neural information processing systems. Landauer, T. K. and S. T. Dumais. 1997. “A Solution to Plato’s Problem: The Latent Semantic Analysis Theory of Acquisition, Induction, and Representation of Knowledge.” Psychological Review; Psychological Review 104(2):211. Landauer, Thomas K., Peter W. Foltz, and Darrell Laham. 1998. “An Introduction to Latent Semantic Analysis.” Discourse Processes 25(2–3):259–84. Larsen, Kai R. and David E. Monarchi. 2004. “A Mathematical Approach to Categorization and Labeling of Qualitative Data: The Latent Categorization Method.” Sociological Methodology 34(1):349–92. Lemaire, Benoît and Guy Denhière. 2006. “Effects of High-Order Co-Occurrences on Word Semantic Similarity.” Current Psychology Letters. Behaviour, Brain & Cognition (18, Vol. 1, 2006). Leopold, Edda. 2005. “On Semantic Spaces.” Pp. 63–86 in LDV Forum, vol. 20. 45

Li, Jia and James Z. Wang. 2004. “Studying Digital Imagery of Ancient Paintings by Mixtures of Stochastic Models.” Image Processing, IEEE Transactions on 13(3):340–53. Liu, Ying, Dengsheng Zhang, Guojun Lu, and Wei-Ying Ma. 2007. “A Survey of ContentBased Image Retrieval with High-Level Semantics.” Pattern Recognition 40(1):262– 82. Lombardi, Thomas Edward. 2005. “The Classification of Style in Fine-Art Painting.” Citeseer. Lu, Qin. 2015. “When Similarity Becomes Opposition: Synonyms and Antonyms Discrimination in DSMs.” P. 41 in IJCoL vol. 1, n. 1 december 2015: Emerging Topics at the First Italian Conference on Computational Linguistics. Accademia University Press. Lund, Kevin and Curt Burgess. 1996. “Producing High-Dimensional Semantic Spaces from Lexical Co-Occurrence.” Behavior Research Methods, Instruments, & Computers 28(2):203–8. Manning, C. and H. Schütze. 1999. Foundations of Statistical Natural Language Processing. Cambridge: MIT Press. Mayaffre, Damon. 2008. “De L’occurrence À L’isotopie. Les Co-Occurrences En Lexicométrie.” Syntaxe & Sémantique (9):53–72. Mehler, Alexander. 2003. “Methodological Aspects of Computational Semiotics.” SEED Journal (Semiotics, Evolution, Energy, and Development) 3(3):71–80. Meunier, J. G. 2014. “Humanités Numériques Ou Computationnelles : Enjeux Herméneutiques.” Sens Public 2014. Meunier, J. G. 2017. “Vers Une Sémiotique Computationnelle” edited by S. Badir, I. Darrault, L. Hébert, P. Michelucci, and É. Trudel. Applied Semiotics / Sémiotique Appliquée (16). Meunier, Jean-Guy. 1989. “Artificial Intelligence and Sign Theory.” Semiotica 77(1–3):43– 64. Michel, J. B. et al. 2011. “Quantitative Analysis of Culture Using Millions of Digitized Books.” Science 331(6014):176–182. Mikolov, Tomas, Wen-tau Yih, and Geoffrey Zweig. 2013. “Linguistic Regularities in Continuous Space Word Representations.” Pp. 746–751 in Hlt-naacl, vol. 13. Mimno, David. 2012. “Computational Historiography: Data Mining in a Century of Classics Journals.” Journal on Computing and Cultural Heritage (JOCCH) 5(1):3. Mitchell, Jeff and Mirella Lapata. 2010. “Composition in Distributional Models of Semantics.” Cognitive Science 34(8):1388–1429. Nadin, Mihai. 2011. “Information and Semiotic Processes The Semiotics of Computation.” Cybernetics & Human Knowing 18(1–2):153–175. 46

Neuman, Yair, Yochai Cohen, and Dan Assaf. 2015. “How Do We Understand the Meaning of Connotations? A Cognitive Computational Model.” Semiotica 2015(205):1–16. Osgood, Charles E. 1952. “The Nature and Measurement of Meaning.” Psychological Bulletin 49(3):197. Osgood, Charles E. 1964. “Semantic Differential Technique in the Comparative Study of Cultures1.” American Anthropologist 66(3):171–200. Osgood, Charles Egerton, George John Suci, and Percy H. Tannenbaum. 1957. The Measurement of Meaning. University of Illinois Press. Pankratius, Victor et al. 2016. “Computer-Aided Discovery: Toward Scientific Insight Generation with Machine Support.” IEEE Intelligent Systems 31(4):3–10. Pincemin, Bénédicte. 1999. “Sémantique Interprétative et Analyses Automatiques de Textes: Que Deviennent Les Sèmes?” Sémiotiques (17):71–120. Rastier, F. 1996. “La Sémantique Des Textes: Concepts et Applications.” Hermes 9(16):15– 37. Rastier, F. 2011. La Mesure et Le Grain. Sémantique de Corpus. Paris: Honoré Champion. Rieger, Burghard B. 1981. “Feasible Fuzzy Semantics. On Some Problems of How to Handle Word Meaning Empirically.” Words, Worlds, and Contexts. New Approaches in Word Semantics (Research in Text Theory 6) 193–209. Rieger, Burghard B. 1983. “Clusters in Semantic Space.” Actes Du Congrès International Informatique et Science Humaines 805–814. Rieger, Burghard B. 1989. “Distributed Semantic Representation of Word Meanings.” Pp. 243–273 in Workshop on Parallel Processing: Logic, Organization, and Technology. Springer. Rieger, Burghard B. 1992. “Fuzzy Computational Semantics.” Pp. 197–217 in Fuzzy Systems. Proceedings of the Japanese-German-Center Symposium, Series, vol. 3. Rieger, Burghard B. 1999. “Semiotics and Computational Linguistics.” Pp. 93–118 in Computing with Words in Information/Intelligent Systems 1. Springer. Sahlgren, M. 2006. “The Word-Space Model: Using Distributional Analysis to Represent Syntagmatic and Paradigmatic Relations between Words in High-Dimensional Vector Spaces.” Stockholm, Stockholm. Sahlgren, Magnus. 2005. “An Introduction to Random Indexing.” in Methods and applications of semantic indexing workshop at the 7th international conference on terminology and knowledge engineering, TKE, vol. 5. Sahlgren, Magnus. 2008. “The Distributional Hypothesis.” Italian Journal of Linguistics 20(1):33–54. Saleh, Babak and Ahmed Elgammal. 2015. “Large-Scale Classification of Fine-Art Paintings: Learning The Right Metric on The Right Feature.” arXiv Preprint arXiv:1505.00855. 47

Santus, Enrico, Alessandro Lenci, Qin Lu, and Sabine Schulte Im Walde. 2014. “Chasing Hypernyms in Vector Spaces with Entropy.” Pp. 38–42 in Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, vol. 2. Santus, Enrico, Qin Lu, Alessandro Lenci, and Churen Huang. 2014. “Unsupervised Antonym-Synonym Discrimination in Vector Space.” Schütze, Hinrich and Jan Pedersen. 1993. “A Vector Model for Syntagmatic and Paradigmatic Relatedness.” Pp. 104–13 in Proceedings of the 9th Annual Conference of the UW Centre for the New OED and Text Research. Citeseer. Shamir, Lior. 2012. “Computer Analysis Reveals Similarities between the Artistic Styles of Van Gogh and Pollock.” Leonardo 45(2):149–54. Shamir, Lior. 2015. “What Makes a Pollock Pollock: A Machine Vision Approach.” International Journal of Arts and Technology 8(1):1–10. Shamir, Lior and Jane A. Tarakhovsky. 2012. “Computer Analysis of Art.” Journal on Computing and Cultural Heritage (JOCCH) 5(2):7. Shen, Jialie. 2009. “Stochastic Modeling Western Paintings for Effective Classification.” Pattern Recognition 42(2):293–301. Shutova, Ekaterina. 2010. “Models of Metaphor in NLP.” Pp. 688–697 in Proceedings of the 48th annual meeting of the association for computational linguistics. Association for Computational Linguistics. Smeulders, Arnold WM, Marcel Worring, Simone Santini, Amarnath Gupta, and Ramesh Jain. 2000. “Content-Based Image Retrieval at the End of the Early Years.” Pattern Analysis and Machine Intelligence, IEEE Transactions on 22(12):1349–80. Stork, David G. 2009. “Computer Vision and Computer Graphics Analysis of Paintings and Drawings: An Introduction to the Literature.” Pp. 9–24 in Computer Analysis of Images and Patterns. Springer. Sylvestre, David, ed. 1997. René Magritte. Catalogue Raisonné. Anvers: Fonds Mercator. Tanaka-Ishii, Kumiko. 2010. Semiotics of Programming. Cambridge University Press. Tanaka-Ishii, Kumiko. 2015. “Semiotics of Computing: Filling the Gap Between Humanity and Mechanical Inhumanity.” Pp. 981–1002 in International Handbook of Semiotics. Springer. Turney, P. D. and P. Pantel. 2010. “From Frequency to Meaning: Vector Space Models of Semantics.” Journal of Artificial Intelligence Research 37(1):141–188. Van Rijsbergen, Cornelis Joost. 2004. The Geometry of Information Retrieval. Cambridge University Press Cambridge. Widdows, Dominic. 2003. “Orthogonal Negation in Vector Spaces for Modelling WordMeanings and Document Retrieval.” Pp. 136–43 in Proceedings of the 41st Annual

48

Meeting on Association for Computational Linguistics-Volume 1. Association for Computational Linguistics. Widdows, Dominic. 2004. Geometry and Meaning. Standford: CSLI Publications. Widdows, Dominic. 2008. “Semantic Vector Products: Some Initial Investigations.” Retrieved September 8, 2016 (https://research.google.com/pubs/pub33477.html). Widdows, Dominic and Trevor Cohen. 2014. “Reasoning with Vectors: A Continuous Model for Fast Robust Inference.” Logic Journal of IGPL jzu028. Zhang, Dengsheng, Md Monirul Islam, and Guojun Lu. 2012. “A Review on Automatic Image Annotation Techniques.” Pattern Recognition 45(1):346–62. Zujovic, Jana, Lisa Gandy, Scott Friedman, Bryan Pardo, and Thrasyvoulos N. Pappas. 2009. “Classifying Paintings by Artistic Genre: An Analysis of Features & Classifiers.” Pp. 1–5 in Multimedia Signal Processing, 2009. MMSP’09. IEEE International Workshop on. IEEE.

49