The role of surprise in theory testing

2 downloads 0 Views 2MB Size Report
Mar 25, 2011 - Computational modeling of music cognition. □ Case study ... Popper, 1963) ... the end of a music performance (e.g., Hudson, 1996;. Clarke ...
The role of surprise in theory testing A case study from music cognition Henkjan Honing ILLC | CSCA University of Amsterdam www.hum.uva.nl/mcg

Friday, 25 March 2011

Thanks to ■ Institute for Logic, Language & Computation (ILLC) ■ Cognitive Science Center Amsterdam (CSCA) ■ Royal Netherlands Academy of Arts and Sciences (KNAW)

2 Friday, 25 March 2011

Outline ■ Computational modeling of music cognition ■ Case study How to select among alternative models? ■ What makes a model surprising? ■ Discussion ■

3 Friday, 25 March 2011

Computational modeling

Machine

4 Friday, 25 March 2011

exhibits

Behavior

agrees

Stimuli

Knowledge

models

Music

Algorithm

Behavior

exhibits

Desain & Honing (1995/2004)

Computational modeling

Machine

5 Friday, 25 March 2011

exhibits

Behavior

agrees

Stimuli

Mental Process

models

Mind

Algorithm

Behavior

exhibits

Desain & Honing (1995/2004)

Computational modeling

Machine

6 Friday, 25 March 2011

exhibits

validates

Algorithm

exhibits exhibits

Behavior

agrees

Stimuli

Mental Process

models

Mind

Behavior

Desain & Honing (1995/2004)

Model selection methods A. Measure of goodness-of-fit (e.g. Rodgers & Rowe, 2002) B. Measure of simplicity (e.g. Pitt, Myung, & Zhang, 2002) C. Measure of surprise (cf. Popper, 1963) D. ‘Ground-truth’ verification (e.g. Rijsbergen, 1979) E. ...

7 Friday, 25 March 2011

Case study ■ Computational models of expressive timing in music performance (e.g., Sundberg & Verillo, 1980;

Kronman & Sundberg, 1987; Longuet-Higgins & Lisle, 1989; Feldman, Epstein & Richards, 1992; Todd, 1992; Epstein, 1994; Todd, 1995; Friberg & Sundberg, 1999; Large & Palmer, 2002)

■ Modeling the Final Ritard: Typical slowing down at

the end of a music performance (e.g., Hudson, 1996; Clarke, 1999; Gabrielsson, 1999)

8 Friday, 25 March 2011

Honing (2006)

Two computational approaches ■ Kinematic approach (K model) Predicts shape of expressive timing patterns and how they conform to the laws of physical motion (commonality)

■ Perception-based approach (P model) Predicts the amount of expressive freedom a performer has in the interpretation of a rhythmic fragment before being ‘misinterpreted’ as an altogether different rhythm (diversity)

9 Friday, 25 March 2011

10 Friday, 25 March 2011

Repp (1992)

K model

11 Friday, 25 March 2011

v(t)=u+at

(1)

v(x)=(u2+2ax)1/2

(2)

v(x)=[1+(wq-1)x]1/q

(3)

Mechanical version of K model

12 Friday, 25 March 2011

Mechanical version of K model

13 Friday, 25 March 2011

www.hum.uva.nl/mmm/fr

Mechanical version of K model

14 Friday, 25 March 2011

P model ■ Two components: - Model of perceived regularity (tempo tracker) (Large & Jones, 1999; Toiviainen, 1999)

- Model of rhythmic categorization (quantizer) (Longuet-Higgins, 1987; Desain & Honing, 1989)

15 Friday, 25 March 2011

Effect of rhythm and tempo on predictions

16 Friday, 25 March 2011

Honing (2005)

A. Measure of good fit

17 Friday, 25 March 2011

Honing (2006)

Conclusion of method A ■ Measures of GOF only assess fit ■ GOF is not able to distinguish between variations in ■

the data caused by noise and those that the model was designed to capture Even if one model would have a significant better fit we could not select that model over the other

18 Friday, 25 March 2011

B. Measure of flexibility ‘Response area’: Range of possible predictions K model P model

19 Friday, 25 March 2011

Honing (2005)

B. Measure of flexibility Effect of note density and rhythmic structure P model

20 Friday, 25 March 2011

Honing (2005)

Conclusion of method B ■ Both models making roughly similar fits to the data ■ K model simpler than the P model ■ However, the P model show less flexibility, and should hence be preferred

21 Friday, 25 March 2011

Conclusion of method B ■ Both models making roughly similar fits to the data ■ K model simpler than the P model ■ However, the P model show less flexibility, and should hence be preferred

■ Still, we can wonder how ‘surprising’ all this is in the context of the phenomenon modeled

22 Friday, 25 March 2011

C. Element of ‘surprise’

23 Friday, 25 March 2011

Honing (2006)

C. Element of ‘surprise’

24 Friday, 25 March 2011

Honing (2006)

Towards a measure of surprise ■ ‘Confirmations [of a theory] should count only if

they are the result of risky predictions; that is to say, if, unenlightened by the theory in question, we should have expected an event which was incompatible with the theory — an event which would have refuted the theory.’ (Popper, 1963:47)

25 Friday, 25 March 2011

Towards a measure of surprise ■ Correct prediction of an unlikely event is more

surprising than the correct prediction of something that was expected anyway

■ Prefer the model that: ■ Minimizes the intersection of Hpredicted with respect to Hplausable ■ While preferring the Hpredicted that is least smooth ■ ...

26 Friday, 25 March 2011

Honing & Romeijn (in prep.)

Towards a measure of surprise Prefer the model that: 1. Fits the empirical data well (best fit) 2. Makes limited range predictions (least flexible) 3. Makes unexpected predictions (most surprising)

27 Friday, 25 March 2011

Epilog Meeting a friend in the corridor, Wittgenstein said: “Tell me, why do people always say it was natural for men to assume that the sun went round the earth rather than that the earth was rotating?” His friend said: “Well, obviously, because it just looks as if the sun is going round the earth.” To which the philosopher replied, “Well, what would it have looked like if it had looked as if the earth was rotating?” 28 Friday, 25 March 2011

Tom Stoppard, Jumpers, 1972

Epilog Meeting a friend in the corridor, Wittgenstein said: “Tell me, why do people always say it was natural for men to assume that the sun went round the earth rather than that the earth was rotating?” His friend said: “Well, obviously, because it just looks as if the sun is going round the earth.” To which the philosopher replied, “Well, what would it have looked like if it had looked as if the earth was rotating?” 29 Friday, 25 March 2011

Tom Stoppard, Jumpers, 1972

References Desain, P., & Honing, H. (2004). Final Report NWO-PIONIER Project "Music, Mind, Machine". Technical Notes ILLC, X-2004-02 http://dare.uva.nl/en/record/117783 Honing, H. (2005). Music Cognition: Theory Testing and Model Selection. Proceedings of the XXVII Annual Conference of the Cognitive Science Society (CogSci2005), 38, Stresa: University of Turin Honing, H. (2006) Computational modeling of music cognition: A case study in model selection. Music Perception, 23(5), 365-376 www.hum.uva.nl/mmm/publications.html Honing, H. (2007) Preferring the best fitting, least flexible, and most surprising prediction: Towards a Bayesian approach to model selection in music cognition. Proceedings of the Society for Music Perception and Cognition (SMPC), 37, Montreal: Concordia University Roberts, S. & Pashler, H. (2000) How persuasive is a good fit? A comment on theory testing. Psychological Review, 107(2), 358-367 Pitt, M.A., Myung, I.J., & Zsang, S. (2002) Toward a method of selecting among computational models of cognition. Psychological Review, 109(3), 472–491 Pitt, M.A., & Myung, I.J. (2002) When a good fit can be bad. Trends in Cognitive Science, 6, 421–425 Popper, K.R. (1963) Conjectures and Refutations:The Growth of Scientific Knowledge. London: Routledge Rodgers, J.L., & Rowe, D.C. (2002) Theory development should begin (but not end) with good empirical fits: A comment on Roberts and Pashler (2000). Psychological Review, 109(3), 599– 604 Van Rijsbergen, C. (1979) Information Retrieval. London: Butterworth

Friday, 25 March 2011