Using R to Assess Mathematical Sense-Making in ... - R Project

Using R to Assess Mathematical Sense-Making in Introductory Physics Courses by Brian Danielak, Eric Kuo, Michael M. Hull, and Andrew Elby

2010 UseR! Conference - July 21, 2010 1

I work behind here

2

Students in conventional courses often treat formulas as algorithms.

Sherin, B. L. (2001). How Students Understand Physics Equations. Cognition and Instruction, 19(4), p. 479 3

if I know the initial velocity...

The uniform acceleration...

and the time...

v = v0 + at Then I can calculate the final velocity!

4

Sense-making is about seeking coherence

5

v = v0 + at

v = v0 + at

v = v0 + at if I know the initial velocity...

t1

The uniform acceleration...

+

=

and the time...

v = v0 + at Then I can calculate the final velocity!

8

v = v0 + at if I know the initial velocity...

t1 t2

The uniform acceleration...

=

+

=

+

and the time...

v = v0 + at Then I can calculate the final velocity!

9

v = v0 + at if I know the initial velocity...

t1 t2 t3

The uniform acceleration...

=

+

=

+

=

+

and the time...

v = v0 + at Then I can calculate the final velocity!

10

The same mathematical structure models lots of other situations

11

money = P + rt t1

=

+

t2

=

+

t3

=

+

12

ˆ Y = β 0 + β 1 X1 x1

=

+

x2

=

+

x3

=

+

13

So here’s the big Question

14

v = v0 + at if I know the initial velocity...

How can we test for thinking like this

t1 t2 t3

The uniform acceleration...

=

+

=

+

=

+

and the time...

v = v0 + at Then I can calculate the final velocity!

15

When we usually test for thinking like this?

if I know the initial velocity...

The uniform acceleration...

v = v0 + at Then I can calculate the final velocity!

10

15

and the time...

A Tale of 3 Instructors

1

2

3

New

Veteran

Veteran

Sense-making Traditional Approach 16

And one Final Exam

Multiple Choice

Free Response

17

And one Final Exam

Multiple Choice

Free Response

18

1 2 3

426 Students 3 Classes

4 5 6 7 8

19

Sense-making items chosen a priori

Class 1

Class 2

3

4

2

e-score

20

Class 3

Looking graphically at the data

21

Across all 8 MC items Raw Score Distributions by Instructor 35 30 25 1

Sense-making (new)

2

Sense-making (vet)

3

Traditional (vet)

20 15 10 5 0 35 30

count

25 20 15 10 5 0 35 30 25 20 15 10 5 0 0

2

4

rawscore

22

6

8

Just the 5 Sense-making Items Escore Distributions by Instructor 40

1

Sense-making (new)

2

Sense-making (vet)

3

30

Traditional (vet)

20 10 0

40

count

30 20 10 0

40 30 20 10 0 −1

0

1

2

3

escore

23

4

5

6

We ran a one-way ANOVA, then MCPs

24

Result 1 Across all 8 Multiple Choice items, Sense-making > traditional approach

25

3−2

3−1

2−1

95% family−wise confidence level

−0.15

−0.10

−0.05

0.00

0.05

Differences in mean levels of instructor Rawratio

26

0.10

Across all 8 MC items 95% Family-wise Confidence Level

2 1

3 1

3 2

TukeyHSD

0.061

-0.019

-0.080

p-adj

0.053

0.751

0.007

Cohen’s d

0.277

-0.0839

-0.365

27

Result 2 Across the 5 specific sense-making items, Sense-making >> traditional approach

28

3−2

3−1

2−1

95% family−wise confidence level

−0.20

−0.15

−0.10

−0.05

0.00

0.05

Differences in mean levels of instructor Eratio

29

0.10

On Sense-making MCs 95% Family-wise Confidence Level

2 1

3 1

3 2

TukeyHSD

0.0450

-0.0871

-0.132

p-adj

0.296

0.0130

< 0.001

Cohen’s d

0.182

-0.331

-0.510

30

Exploring Free-Response Data

31

What happens to mechanical energy after the explosion?

?

32

What Happens to the Mechanical Energy of the Two Blocks? 140

120

100 q09d stays the same 80

count

increases blank decreases

60

missing value uncodeable

40

20

0 1

2

instructor

3

33

What Happens to the Mechanical Energy of the Two Blocks? 140

120

100 q09d stays the same 80

count

increases blank decreases

60

missing value uncodeable

40

20

0 1

2

instructor

3

34

What Happens to the Mechanical Energy of the Two Blocks? 140

120

100 q09d stays the same 80

count

increases blank decreases

60

missing value uncodeable

72

40

20

43

43

1

2

0

instructor

3

35

What Happens to the Mechanical Energy of the Two Blocks? 140

120

100

Mechanical Energy Stays the Same

q09d stays the same

80

count

increases blank decreases

60

missing value uncodeable

72

40

20

43

43

0 1

2

instructor

3

36

Mechanical Energy Stays the Same

72 43

43

Sense-making (new)

Sense-making (vet)

= Traditional (vet) 37

Mechanical Energy Stays the Same

72 43

43

Sense-making (new)

Sense-making (vet)

= Traditional (vet) 38

Result 3 When considering energy, more traditional students asserted something counter-intuitive

72 43

43

Sense-making (new)

Sense-making (vet)

39

Traditional (vet)

What Happens to the Mechanical Energy of the Two Blocks? 140

120

100 q09d stays the same 80

count

increases blank decreases

60

missing value uncodeable

40

20

0 1

2

instructor

3

40

Mechanical Energy Increases

59

33

75

41

Mechanical Energy Increases

59

33

75

42

Mechanical Energy Increases

59

75 33

43

Mechanical Energy Increases

59

75 33

Sense-making (new)

Sense-making (vet)

>

Traditional (vet) 44

Mechanical Energy Increases

59

75 33

Sense-making (new)

Sense-making (vet)

>

Traditional (vet) 45

Result 4 When considering energy, more sense-making students were consistent with simple intuition.

59

75 33

Sense-making (new)

46

Sense-making (vet)

Traditional (vet)

Ahead,

!ar b"...

DRAGONS!

http://www.flickr.com/photos/aztlek/2421800480/in/ 47 set-72157605762980035/

This stuff’s in beta.

http://www.flickr.com/photos/aztlek/2421800480/in/ 48 set-72157605762980035/

Students Who Quoted ‘‘Conservation of Energy'' as their #§* Explanation for q09d

30

25

count

20 variable 5

15

5x

10

5

0 1

2

instructor 49

3

Mechanical Energy Stays the Same...

32

8

11

50

Mechanical Energy Stays the Same... Because Energy is Conserved

8 Sense-making (new)

32

11 Sense-making (vet) 51

Traditional (vet)

Result 5 32

When justifying their answer, more traditional students just quoted a rule, ignoring that it was tautological and inaccurate.

8

Sense-making (new) 52

11 Sense-making (vet)

Traditional (vet)

Summary

53

95% family−wise confidence level 2−1

Result 3 72 43

43

Sense-making (new)

Sense-making (vet)

Traditional (vet)

3−2

3−1

When considering energy, more traditional students asserted something counter-intuitive

−0.20

−0.15

−0.10

−0.05

0.00

0.05

0.10

Differences in mean levels of instructor Eratio

29

39

Result 4

Result 5

When considering energy, more sense-making students were consistent with simple intuition.

59

75 33

Sense-making (new)

Sense-making (vet)

32

When justifying their answer, more traditional students just quoted a rule, ignoring that it was tautological and inaccurate.

8

Traditional (vet)

Sense-making (new) 46

54

52

11 Sense-making (vet)

Traditional (vet)

http://github.com/briandk/user2010public

55

Thanks! Project Directors

Andy Elby

Ayush Gupta

Graduate Assistants

Eric Kuo

Mike Hull 56

Many Thanks

NSF-DRL grant # 0733613 NSF-EEC grant # 0835880 57

95% family−wise confidence level 2−1

Result 3 72 43

43

Sense-making (new)

Sense-making (vet)

Traditional (vet)

3−2

3−1

When considering energy, more traditional students asserted something counter-intuitive

−0.20

−0.15

−0.10

−0.05

0.00

0.05

0.10

Differences in mean levels of instructor Eratio

29

39

Result 4

Result 5

When considering energy, more sense-making students were consistent with simple intuition.

59

75 33

Sense-making (new)

Sense-making (vet)

32

When justifying their answer, more traditional students just quoted a rule, ignoring that it was tautological and inaccurate.

8

Traditional (vet)

Sense-making (new) 46

58

52

11 Sense-making (vet)

Traditional (vet)