Symbolic music comparison with tree data structures

MaMuX March, 2012

David Rizo Univ. Alicante

Metrical trees music comparison, classification and composition

Trees & Science

Newton Tree

Trees & Music

This talk is about the mathematical structure TREE and MUSIC

HUMAN MUSICAL PROCESS

Creation


Based on an abstraction / structure Creation


Based on an abstraction / structure Symbols: score Creation


Based on an abstraction / structure Symbols: score Creation Interpretation: produces


Based on an abstraction / structure Symbols: score Creation Music Sound

Interpretation: produces

COMPUTER MUSICAL PROCESS

Creation


Creation

Based on an abstraction / structure


Creation


Symbols: score


Creation


Symbols: score



Creation

Music Sound


Symbols: score



Analysis

Music Sound


Symbols: score


What role can trees play in music ?


What role can trees play in music ?

What role can trees play in computer music ?

REPRESENTATION • Internal

intermediate representation dramatically conditions the application


intermediate representation dramatically conditions the application • e.g. Which

is better for printing a score?



a) Sequence of frequency and onset time in milliseconds






b) Sequence of note names and figures





b) Sequence of note names and figures

REPRESENTATION

REPRESENTATION • Which

is best


is best

• depends

on the application


is best

• depends

on the application

• Composition


is best

• depends

on the application

• Composition • easily

generate new content


is best

• depends

on the application



• Analysis

and retrieval


is best

• depends

on the application



• Analysis • easily

and retrieval

capture important features

SYMBOLIC REPRESENTATIONS • Strings • n-grams • Graphs • Point

sets

• Geometric • Hidden • Spiral

Markov Models

arrays, etc....

• Trees

_

ABSTRACT DATA TYPE TREE B4

• Suitable E5 B4 h

h

I

• music V

fT

fD

AM

_

_

D5

A4

intermediate representation for: B4 D5 D5 D5 A4 h

h

h

h

information (MIR) C#5 V V retrieval V V

AM • composing

B4

C#5

h

n

V

V

fD

fD

AM

AM

A4

h

h

VI

VI

fD

fD

fD

fD

fT

fT

AM

AM

AM

AM

AM

AM

OVERVIEW •

Background •

•

•

Intermediate representations The origin: strings •

•

Previous uses of trees in computer music

Our tree approach •

metric tree construction

•

probabilistic k-testables

Applications •

classification and generation

•

composition

Strings | Tree state of the art | Construction | Applications | Conclusions

String = sequence of symbols

STRINGS n1 •A

n2

n3

n4

···

nm

melody is represented as a sequence of symbols ni

• each

symbol ni may represent a note

! 2 " 4 ! !

Pitch Rhythm

STRINGS n1 •A

n2

n3

···

n4

nm


• each


! 2 " 4 ! !

Pitch Rhythm

G

C

B

Note name

STRINGS n1 •A

n2

n3

n4

···

nm


• each


! 2 " 4 ! !

Pitch Rhythm

67

72

71

MIDI pitch

STRINGS n1 •A

n2

n3

n4

···

nm


• each


! 2 " 4 ! !

Pitch

67

72

71

MIDI pitch

Rhythm

1

1/2

1/2

Duration

STRINGS n1 •A

n2

n3

···

n4

nm


• each


! 2 " 4 ! !

Pitch

67

72

71

MIDI pitch

Rhythm

0

1

3/2

Onset

STRINGS n1 •A

n2

n3

···

n4

nm


• each


! 2 " 4 ! !

Pitch

67

72

71

MIDI pitch

Rhythm

0

1

3/2

Onset

G,0

C,1

B,3/2

A combination

[Pitch, Onset]

STRINGS n1 •A

n2

n3

n4

···

nm


• each

symbol ni may represent a note or a relation 4th

! 2 " 4 ! !

2nd

interval

STRINGS n1 •A

n2

n3

···

n4

nm


• each

symbol ni may represent a note or a relation

! 2 " 4 ! !

4th

2nd

interval

STRINGS n1 •A

n2

n3

n4

···

nm


• each

symbol ni may represent a note or a relation UP

! 2 " 4 ! !

DOWN

contour

STRINGS n1 •A

n2

n3

n4

···

nm


• each


! 2 " 4 ! !

UP

DOWN

contour

DECOUPLED STRINGS n1 •A

n2

n3

···

n4

nm


• each


G

0

C

1

B

3/2

onset

pitch

onset

pitch

onset

" 42 ! ! !

pitch

any of these dimensions

STRING LANGUAGES

STRING LANGUAGES • String

= sequence of symbols

STRING LANGUAGES • String • Each


symbol belongs to an alphabet ∑




• String

= sequence of ∑




• String

= sequence of ∑

• denoted

by ∑* (meaning: 0 or more symbols)




• String

= sequence of ∑

• denoted • ∑+

by ∑* (meaning: 0 or more symbols)

denotes 1 or more symbols

STRING LANGUAGES

STRING LANGUAGES • Language

= set of strings


= set of strings

automaton can generate and recognize this kind of language following states and transitions

• An


= set of strings


• An

• e.g. Let

∑ = {A, B, C},


= set of strings


• An

• e.g. Let

∑ = {A, B, C},

the language AB+C is the set of strings


= set of strings


• An

• e.g. Let

∑ = {A, B, C},

the language AB+C is the set of strings starting with A A


= set of strings


• An

• e.g. Let

∑ = {A, B, C},

the language AB+C is the set of strings starting with A

containing 1 or more B A

B


= set of strings


• An

• e.g. Let

∑ = {A, B, C},

the language AB+C is the set of strings starting with A

containing 1 or more B A

B

C

ending with C

STRING LANGUAGES

STRING LANGUAGES •A

music language can be built on strings



• if

only notes are represented,



• if

only notes are represented, ∑ represents a combination pitch (∑p) and rhythm (∑r) ∑ = ∑p ⊗ ∑r

• alphabet



• if


• alphabet

• Coupled

∑ = ∑p x ∑r



• if


• alphabet

• Coupled

∑ = ∑p x ∑r

• Decoupled

∑ = ∑p U ∑r

PITCH ALPHABETS ∑p Absolute magnitudes ! Scientific pitch:

F4

C 5 D5 D5

G4

R

G4 A4 B4

! Pitch class:

5

0

2

2

7

R

7

9 11

! Contour:

x

+

+

=

-

=

+

! High Def. Contour

x

+2 +1

0

-2

0

+1 +1

! Intervals:

x

+7 +2

0

-7

0

+2 +2

! Relative to tonic:

5

2

7

Relative magnitudes

0

2

etc.....

R

7

9

+

11

|!p| = 89 |!p| = 13 |!p| = 3 |!p| = 5 |!p| = 49 |!p| = 13

RHYTHM ALPHABETS ∑r Absolute magnitudes ! Durations: n c c n ! Inter-onset intervals (IOIs): 1 1/2 1/2 1 Relative magnitudes • Duration relative to previous x 1/2 1 • Duration relative to most frequent 2 1 1 • Relation between IOIs (IORs): 1/2 1 2 ! Rhythmic contour: x - =

2

n

n

2

c

b

1/2 1/2 (2) IOIn = on+1 - on

1

1

2

2

2

2 +

1/4 =

=

etc.....

c

1/2 1 1

1

4

4 IORn = IOIn+1/IOIn 1 4 x - = +

MONOPHONIC

POLYPHONIC

STRINGS - POLYPHONY • Chord

sequence approach:

Onset group based


sequence approach:

Onset group based requires segmentation

∑ = (∑r x ∑+p )

{C,F,A}, 0

{G,D,F,B}, 2

{C,E,G,B}, 4


sequence approach:

Onset group based requires segmentation

{C,F,A}, 0

{G,D,F,B}, 2

{C,E,G,B}, 4

∑ = (∑r x ∑+p ) Named chord: requires chord analysis ∑ = set of chord names

IV

V

I

STRINGS - POLYPHONY • Individual

note approach

Not interleaved: requires voice analysis, deal with divisi, etc...


note approach

Not interleaved: requires voice analysis, deal with divisi, etc... sequential arrangement of all monophonic lines


note approach


A

B

C


note approach


A

B

C

F

F

G


note approach


A

B

C

F

F

G

C

D

E


note approach


A

B

C

F

F

G

C

D

E

G

C


note approach


∑ = note alphabet + voice

A

B

C

F

F

G

C

D

E

G

C


note approach

Interleaved


note approach

Interleaved


note approach

Interleaved


note approach

Interleaved notes sorted lexicographically (e.g. first order by onset, them by pitch)


note approach


C,0

F,0

A,0

G,2

D,2

F2

B,2

C,4

E,4

G,4

C4


note approach


∑ = ∑r x ∑p

C,0

F,0

A,0

G,2

D,2

F2

B,2

C,4

E,4

G,4

C4

Evolve Strings to Trees

Evolve Strings to Trees • Many

possible note representations (many alphabets ∑)

Evolve Strings to Trees • Many

possible note representations (many alphabets ∑)

• But...

difficult to represent structure

String

n1

n2

n3

n4

···

In strings each note /symbol is linked to | from just other note/symbol

nm

String

n1

n2

n3

n4

···

nm

In strings each note /symbol is linked to | from just other note/symbol If some structure is required? e.g. n1

n2

n3

n4

···

nm

STRUCTURE • e.g. “Alberti

C

G

E

G

Bass” like piano left hand accompaniment

C

G

E

G

D

G

F

G

C

G

E

G


C

G

E

G


C

G

E

G

D

G

F

G

C

G

E

G



C

I

C

G

E

G

G

E

G

D

G

F

G

C

G

E

G



I

C

G

E

G

I

C

G

V7

E

G

D

G

I

F

G

C

G

E

G



I

C

G

E

G

I

C

G

V7

E

G

D

G

I

F

G

C

G

E

G

STRUCTURE

I

C

G

E

G

I

C

G

V7

E

G

D

G

F

I

G

C

G

E

G

STRUCTURE Production: Rewrite I as C G E G

I

C

G

E

G

I

C

G

V7

E

G

D

G

F

I

G

C

G

E

G


I

C

G

E

G

I →C G E G

I

C

G

V7

E

G

D

G

F

I

G

C

G

E

G


I

C

G

E

G

I →C G E G

I

C

G

V7

E

G

D

G

F

I

G

C

G

E

G


I →C G E G V7 →D G F G

I

C

G

E

G

I

C

G

V7

E

G

D

G

F

I

G

C

G

E

G



I

C

G

E

G

I

C

G

V7

E

G

D

G

F

I

G

C

G

E

G

STRUCTURE Melody → I I V7 I

Production: Rewrite I as C G E G


I

C

G

E

G

I

C

G

V7

E

G

D

G

F

I

G

C

G

E

G



I →C G E G

Terminals

V7 →D G F G

I

C

G

E

G

I

C

G

V7

E

G

D

G

F

I

G

C

G

E

G



I →C G E G

Terminals

V7 →D G F G

I

C

G

E

G

I

C

G

V7

E

G

D

G

F

I

G

C

G

E

G

∈∑

GRAMMAR Melody → I I V7 I


I →C G E G

Terminals

V7 →D G F G

I

C

G

E

G

I

C

G

V7

E

G

D

G

F

I

G

C

G

E

G

∈∑

GRAMMAR Melody → I I V7 I


I →C G E G

Terminals

V7 →D G F G Parsing tree I

C

G

E

G

I

C

G

V7

E

G

D

G

F

I

G

C

G

E

G

∈∑

GRAMMAR • Generalizing

the grammar

Melody → ChordRealization+ ChordRealization → Note Note Note Note Note ∈ any ∑

TREE PROPERTY

Trees capture more easily structure than strings

Strings | Tree state of the art | Construction | Applications | Conclusions

MUSIC TREE REPRESENTATION previous uses Formal architecture Grammars Composition Analysis

MUSIC TREE REPRESENTATION Formal architecture

Musical form Grammars Song

1.Period

Composition O-Phrase

1.Motive

C-Phrase

Analysis

3.Period

2.Period

O-Phrase

2.Motive

C-Phrase

O-Phrase

1.Motive

C-Phrase

MUSIC TREE REPRESENTATION Lee’85 grammars

CHAPTER 3. MUSIC SIMILARITY WITH TREES

Formal architecture

$ # # #( # #

"!% ' &

Explain rhythm perception with grammars (a) First two bars of“Auld Lang Syne”

Grammars Composition

S→ ¯ |