Symbolic music comparison with tree data structures

5 downloads 122 Views 15MB Size Report
This talk is about the mathematical structure. TREE and. MUSIC .... Strings | Tree state of the art | Construction | Applications | Conclusions ...
MaMuX March, 2012

David Rizo Univ. Alicante

Metrical trees music comparison, classification and composition

Trees & Science

Newton Tree

Trees & Music

This talk is about the mathematical structure TREE and MUSIC

HUMAN MUSICAL PROCESS

Creation

HUMAN MUSICAL PROCESS

Based on an abstraction / structure Creation

HUMAN MUSICAL PROCESS

Based on an abstraction / structure Symbols: score Creation

HUMAN MUSICAL PROCESS

Based on an abstraction / structure Symbols: score Creation Interpretation: produces

HUMAN MUSICAL PROCESS

Based on an abstraction / structure Symbols: score Creation Music Sound

Interpretation: produces

COMPUTER MUSICAL PROCESS

Creation

COMPUTER MUSICAL PROCESS

Creation

Based on an abstraction / structure

COMPUTER MUSICAL PROCESS

Creation

Based on an abstraction / structure

Symbols: score

COMPUTER MUSICAL PROCESS

Creation

Based on an abstraction / structure

Symbols: score

Interpretation: produces

COMPUTER MUSICAL PROCESS

Creation

Music Sound

Based on an abstraction / structure

Symbols: score

Interpretation: produces

COMPUTER MUSICAL PROCESS

Analysis

Music Sound

Based on an abstraction / structure

Symbols: score

Interpretation: produces

What role can trees play in music ?

Based on an abstraction / structure

What role can trees play in music ?

What role can trees play in computer music ?

REPRESENTATION • Internal

intermediate representation dramatically conditions the application

REPRESENTATION • Internal

intermediate representation dramatically conditions the application • e.g. Which

is better for printing a score?

REPRESENTATION • Internal

intermediate representation dramatically conditions the application • e.g. Which

a) Sequence of frequency and onset time in milliseconds

is better for printing a score?

REPRESENTATION • Internal

intermediate representation dramatically conditions the application • e.g. Which

a) Sequence of frequency and onset time in milliseconds

is better for printing a score?

b) Sequence of note names and figures

REPRESENTATION • Internal

intermediate representation dramatically conditions the application • e.g. Which

a) Sequence of frequency and onset time in milliseconds

is better for printing a score?

b) Sequence of note names and figures

REPRESENTATION

REPRESENTATION • Which

is best

REPRESENTATION • Which

is best

• depends

on the application

REPRESENTATION • Which

is best

• depends

on the application

• Composition

REPRESENTATION • Which

is best

• depends

on the application

• Composition • easily

generate new content

REPRESENTATION • Which

is best

• depends

on the application

• Composition • easily

generate new content

• Analysis

and retrieval

REPRESENTATION • Which

is best

• depends

on the application

• Composition • easily

generate new content

• Analysis • easily

and retrieval

capture important features

SYMBOLIC REPRESENTATIONS • Strings • n-grams • Graphs • Point

sets

• Geometric • Hidden • Spiral

Markov Models

arrays, etc....

• Trees

_

ABSTRACT DATA TYPE TREE B4

• Suitable E5 B4 h

h

I

• music V

fT

fD

AM

_

_

D5

A4

intermediate representation for: B4 D5 D5 D5 A4 h

h

h

h

information (MIR) C#5 V V retrieval V V

AM • composing

B4

C#5

h

n

V

V

fD

fD

AM

AM

A4

h

h

VI

VI

fD

fD

fD

fD

fT

fT

AM

AM

AM

AM

AM

AM

OVERVIEW •

Background •





Intermediate representations The origin: strings •



Previous uses of trees in computer music

Our tree approach •

metric tree construction



probabilistic k-testables

Applications •

classification and generation



composition

Strings | Tree state of the art | Construction | Applications | Conclusions

String = sequence of symbols

STRINGS n1 •A

n2

n3

n4

···

nm

melody is represented as a sequence of symbols ni

• each

symbol ni may represent a note

! 2 " 4 ! !

Pitch Rhythm

STRINGS n1 •A

n2

n3

···

n4

nm

melody is represented as a sequence of symbols ni

• each

symbol ni may represent a note

! 2 " 4 ! !

Pitch Rhythm

G

C

B

Note name

STRINGS n1 •A

n2

n3

n4

···

nm

melody is represented as a sequence of symbols ni

• each

symbol ni may represent a note

! 2 " 4 ! !

Pitch Rhythm

67

72

71

MIDI pitch

STRINGS n1 •A

n2

n3

n4

···

nm

melody is represented as a sequence of symbols ni

• each

symbol ni may represent a note

! 2 " 4 ! !

Pitch

67

72

71

MIDI pitch

Rhythm

1

1/2

1/2

Duration

STRINGS n1 •A

n2

n3

···

n4

nm

melody is represented as a sequence of symbols ni

• each

symbol ni may represent a note

! 2 " 4 ! !

Pitch

67

72

71

MIDI pitch

Rhythm

0

1

3/2

Onset

STRINGS n1 •A

n2

n3

···

n4

nm

melody is represented as a sequence of symbols ni

• each

symbol ni may represent a note

! 2 " 4 ! !

Pitch

67

72

71

MIDI pitch

Rhythm

0

1

3/2

Onset

G,0

C,1

B,3/2

A combination

[Pitch, Onset]

STRINGS n1 •A

n2

n3

n4

···

nm

melody is represented as a sequence of symbols ni

• each

symbol ni may represent a note or a relation 4th

! 2 " 4 ! !

2nd

interval

STRINGS n1 •A

n2

n3

···

n4

nm

melody is represented as a sequence of symbols ni

• each

symbol ni may represent a note or a relation

! 2 " 4 ! !

4th

2nd

interval

STRINGS n1 •A

n2

n3

n4

···

nm

melody is represented as a sequence of symbols ni

• each

symbol ni may represent a note or a relation UP

! 2 " 4 ! !

DOWN

contour

STRINGS n1 •A

n2

n3

n4

···

nm

melody is represented as a sequence of symbols ni

• each

symbol ni may represent a note or a relation

! 2 " 4 ! !

UP

DOWN

contour

DECOUPLED STRINGS n1 •A

n2

n3

···

n4

nm

melody is represented as a sequence of symbols ni

• each

symbol ni may represent a note or a relation

G

0

C

1

B

3/2

onset

pitch

onset

pitch

onset

" 42 ! ! !

pitch

any of these dimensions

STRING LANGUAGES

STRING LANGUAGES • String

= sequence of symbols

STRING LANGUAGES • String • Each

= sequence of symbols

symbol belongs to an alphabet ∑

STRING LANGUAGES • String • Each

= sequence of symbols

symbol belongs to an alphabet ∑

• String

= sequence of ∑

STRING LANGUAGES • String • Each

= sequence of symbols

symbol belongs to an alphabet ∑

• String

= sequence of ∑

• denoted

by ∑* (meaning: 0 or more symbols)

STRING LANGUAGES • String • Each

= sequence of symbols

symbol belongs to an alphabet ∑

• String

= sequence of ∑

• denoted • ∑+

by ∑* (meaning: 0 or more symbols)

denotes 1 or more symbols

STRING LANGUAGES

STRING LANGUAGES • Language

= set of strings

STRING LANGUAGES • Language

= set of strings

automaton can generate and recognize this kind of language following states and transitions

• An

STRING LANGUAGES • Language

= set of strings

automaton can generate and recognize this kind of language following states and transitions

• An

• e.g. Let

∑ = {A, B, C},

STRING LANGUAGES • Language

= set of strings

automaton can generate and recognize this kind of language following states and transitions

• An

• e.g. Let

∑ = {A, B, C},

the language AB+C is the set of strings

STRING LANGUAGES • Language

= set of strings

automaton can generate and recognize this kind of language following states and transitions

• An

• e.g. Let

∑ = {A, B, C},

the language AB+C is the set of strings starting with A A

STRING LANGUAGES • Language

= set of strings

automaton can generate and recognize this kind of language following states and transitions

• An

• e.g. Let

∑ = {A, B, C},

the language AB+C is the set of strings starting with A

containing 1 or more B A

B

STRING LANGUAGES • Language

= set of strings

automaton can generate and recognize this kind of language following states and transitions

• An

• e.g. Let

∑ = {A, B, C},

the language AB+C is the set of strings starting with A

containing 1 or more B A

B

C

ending with C

STRING LANGUAGES

STRING LANGUAGES •A

music language can be built on strings

STRING LANGUAGES •A

music language can be built on strings

• if

only notes are represented,

STRING LANGUAGES •A

music language can be built on strings

• if

only notes are represented, ∑ represents a combination pitch (∑p) and rhythm (∑r) ∑ = ∑p ⊗ ∑r

• alphabet

STRING LANGUAGES •A

music language can be built on strings

• if

only notes are represented, ∑ represents a combination pitch (∑p) and rhythm (∑r) ∑ = ∑p ⊗ ∑r

• alphabet

• Coupled

∑ = ∑p x ∑r

STRING LANGUAGES •A

music language can be built on strings

• if

only notes are represented, ∑ represents a combination pitch (∑p) and rhythm (∑r) ∑ = ∑p ⊗ ∑r

• alphabet

• Coupled

∑ = ∑p x ∑r

• Decoupled

∑ = ∑p U ∑r

PITCH ALPHABETS ∑p Absolute magnitudes !   Scientific pitch:

F4

C 5 D5 D5

G4

R

G4 A4 B4

!   Pitch class:

5

0

2

2

7

R

7

9 11

!   Contour:

x

+

+

=

-

=

+

!   High Def. Contour

x

+2 +1

0

-2

0

+1 +1

!   Intervals:

x

+7 +2

0

-7

0

+2 +2

!   Relative to tonic:

5

2

7

Relative magnitudes

0

2

etc.....

R

7

9

+

11

|!p| = 89 |!p| = 13 |!p| = 3 |!p| = 5 |!p| = 49 |!p| = 13

RHYTHM ALPHABETS ∑r Absolute magnitudes !   Durations: n c c n !   Inter-onset intervals (IOIs): 1 1/2 1/2 1 Relative magnitudes •  Duration relative to previous x 1/2 1 •  Duration relative to most frequent 2 1 1 •  Relation between IOIs (IORs): 1/2 1 2 !   Rhythmic contour: x - =

2

n

n

2

c

b

1/2 1/2 (2) IOIn = on+1 - on

1

1

2

2

2

2 +

1/4 =

=

etc.....

c

1/2 1 1

1

4

4 IORn = IOIn+1/IOIn 1 4 x - = +

MONOPHONIC

POLYPHONIC

STRINGS - POLYPHONY • Chord

sequence approach:

Onset group based

STRINGS - POLYPHONY • Chord

sequence approach:

Onset group based requires segmentation

∑ = (∑r x ∑+p )

{C,F,A}, 0

{G,D,F,B}, 2

{C,E,G,B}, 4

STRINGS - POLYPHONY • Chord

sequence approach:

Onset group based requires segmentation

{C,F,A}, 0

{G,D,F,B}, 2

{C,E,G,B}, 4

∑ = (∑r x ∑+p ) Named chord: requires chord analysis ∑ = set of chord names

IV

V

I

STRINGS - POLYPHONY • Individual

note approach

Not interleaved: requires voice analysis, deal with divisi, etc...

STRINGS - POLYPHONY • Individual

note approach

Not interleaved: requires voice analysis, deal with divisi, etc... sequential arrangement of all monophonic lines

STRINGS - POLYPHONY • Individual

note approach

Not interleaved: requires voice analysis, deal with divisi, etc... sequential arrangement of all monophonic lines

A

B

C

STRINGS - POLYPHONY • Individual

note approach

Not interleaved: requires voice analysis, deal with divisi, etc... sequential arrangement of all monophonic lines

A

B

C

F

F

G

STRINGS - POLYPHONY • Individual

note approach

Not interleaved: requires voice analysis, deal with divisi, etc... sequential arrangement of all monophonic lines

A

B

C

F

F

G

C

D

E

STRINGS - POLYPHONY • Individual

note approach

Not interleaved: requires voice analysis, deal with divisi, etc... sequential arrangement of all monophonic lines

A

B

C

F

F

G

C

D

E

G

C

STRINGS - POLYPHONY • Individual

note approach

Not interleaved: requires voice analysis, deal with divisi, etc... sequential arrangement of all monophonic lines

∑ = note alphabet + voice

A

B

C

F

F

G

C

D

E

G

C

STRINGS - POLYPHONY • Individual

note approach

Interleaved

STRINGS - POLYPHONY • Individual

note approach

Interleaved

STRINGS - POLYPHONY • Individual

note approach

Interleaved

STRINGS - POLYPHONY • Individual

note approach

Interleaved notes sorted lexicographically (e.g. first order by onset, them by pitch)

STRINGS - POLYPHONY • Individual

note approach

Interleaved notes sorted lexicographically (e.g. first order by onset, them by pitch)

C,0

F,0

A,0

G,2

D,2

F2

B,2

C,4

E,4

G,4

C4

STRINGS - POLYPHONY • Individual

note approach

Interleaved notes sorted lexicographically (e.g. first order by onset, them by pitch)

∑ = ∑r x ∑p

C,0

F,0

A,0

G,2

D,2

F2

B,2

C,4

E,4

G,4

C4

Evolve Strings to Trees

Evolve Strings to Trees • Many

possible note representations (many alphabets ∑)

Evolve Strings to Trees • Many

possible note representations (many alphabets ∑)

• But...

difficult to represent structure

String

n1

n2

n3

n4

···

In strings each note /symbol is linked to | from just other note/symbol

nm

String

n1

n2

n3

n4

···

nm

In strings each note /symbol is linked to | from just other note/symbol If some structure is required? e.g. n1

n2

n3

n4

···

nm

STRUCTURE • e.g. “Alberti

C

G

E

G

Bass” like piano left hand accompaniment

C

G

E

G

D

G

F

G

C

G

E

G

STRUCTURE • e.g. “Alberti

C

G

E

G

Bass” like piano left hand accompaniment

C

G

E

G

D

G

F

G

C

G

E

G

STRUCTURE • e.g. “Alberti

Bass” like piano left hand accompaniment

C

I

C

G

E

G

G

E

G

D

G

F

G

C

G

E

G

STRUCTURE • e.g. “Alberti

Bass” like piano left hand accompaniment

I

C

G

E

G

I

C

G

V7

E

G

D

G

I

F

G

C

G

E

G

STRUCTURE • e.g. “Alberti

Bass” like piano left hand accompaniment

I

C

G

E

G

I

C

G

V7

E

G

D

G

I

F

G

C

G

E

G

STRUCTURE

I

C

G

E

G

I

C

G

V7

E

G

D

G

F

I

G

C

G

E

G

STRUCTURE Production: Rewrite I as C G E G

I

C

G

E

G

I

C

G

V7

E

G

D

G

F

I

G

C

G

E

G

STRUCTURE Production: Rewrite I as C G E G

I

C

G

E

G

I →C G E G

I

C

G

V7

E

G

D

G

F

I

G

C

G

E

G

STRUCTURE Production: Rewrite I as C G E G

I

C

G

E

G

I →C G E G

I

C

G

V7

E

G

D

G

F

I

G

C

G

E

G

STRUCTURE Production: Rewrite I as C G E G

I →C G E G V7 →D G F G

I

C

G

E

G

I

C

G

V7

E

G

D

G

F

I

G

C

G

E

G

STRUCTURE Production: Rewrite I as C G E G

I →C G E G V7 →D G F G

I

C

G

E

G

I

C

G

V7

E

G

D

G

F

I

G

C

G

E

G

STRUCTURE Melody → I I V7 I

Production: Rewrite I as C G E G

I →C G E G V7 →D G F G

I

C

G

E

G

I

C

G

V7

E

G

D

G

F

I

G

C

G

E

G

STRUCTURE Melody → I I V7 I

Production: Rewrite I as C G E G

I →C G E G

Terminals

V7 →D G F G

I

C

G

E

G

I

C

G

V7

E

G

D

G

F

I

G

C

G

E

G

STRUCTURE Melody → I I V7 I

Production: Rewrite I as C G E G

I →C G E G

Terminals

V7 →D G F G

I

C

G

E

G

I

C

G

V7

E

G

D

G

F

I

G

C

G

E

G

∈∑

GRAMMAR Melody → I I V7 I

Production: Rewrite I as C G E G

I →C G E G

Terminals

V7 →D G F G

I

C

G

E

G

I

C

G

V7

E

G

D

G

F

I

G

C

G

E

G

∈∑

GRAMMAR Melody → I I V7 I

Production: Rewrite I as C G E G

I →C G E G

Terminals

V7 →D G F G Parsing tree I

C

G

E

G

I

C

G

V7

E

G

D

G

F

I

G

C

G

E

G

∈∑

GRAMMAR • Generalizing

the grammar

Melody → ChordRealization+ ChordRealization → Note Note Note Note Note ∈ any ∑

TREE PROPERTY

Trees capture more easily structure than strings

Strings | Tree state of the art | Construction | Applications | Conclusions

MUSIC TREE REPRESENTATION previous uses Formal architecture Grammars Composition Analysis

MUSIC TREE REPRESENTATION Formal architecture

Musical form Grammars Song

1.Period

Composition O-Phrase

1.Motive

C-Phrase

Analysis

3.Period

2.Period

O-Phrase

2.Motive

C-Phrase

O-Phrase

1.Motive

C-Phrase

MUSIC TREE REPRESENTATION Lee’85 grammars

CHAPTER 3. MUSIC SIMILARITY WITH TREES

Formal architecture

$ # # #( # #

"!% ' &

Explain rhythm perception with grammars (a) First two bars of“Auld Lang Syne”

Grammars Composition

S→ ¯ |