Using experience sampling method data for evaluating learning activities Julia Dietrich, Julia Moeller, Jaana Viljaranta, & Bärbel Kracke J. Dietrich, Friedrich-Schiller-University Jena, Germany
Dr. Julia Dietrich 1 / 24
Aims of this talk 1. Introduce an assessment design to evaluate learning activities 2. Discuss statistical coefficients that could be used in the context of motivational interventions at the university level
Comments are welcome!
2 / 24
Motivational Framework of this Study: Expectancy-Value Theory Expectancies and values regarding a task are central motivational forces in students’ academic behaviors and learning. (Eccles et al., 1983; Eccles & Wigfield, 2002)
They predict academic choices, persistence, and achievement.
Expectancies
• Competence beliefs • Expectations for success
Achievement behaviors and choices
(e.g., Battle & Wigfield, 2003; Cole et al., 2008; Durik et al., 2006; Liem et al., 2008; Marsh et al., 2005; Meece et al., 1990; Nagengast et al., 2011; Wigfield et al., 1997)
Learning situations vary in their motivational characteristics, such as their interestingness or the extent to which they make students feel competent. (e.g., Dietrich et al., 2017; Malmberg et al., 2013).
Task values • • • •
Interest/Intrinsic Attainment Utility Cost
Note. Focus-variables of this study in gold.
How motivating is a learning situation? Using the experience sampling method (ESM) to evaluate the motivational characteristics of concrete learning situations (activities, tasks etc.) as opposed to an entire course or subject.
Interesting Diskussion stuff! I don‘t get Zwei Drittel der Lernsituationen this at all.werden konsistent erlebt (gleichsam hohe/geringe Erwartungen und Werte mit geringen/hohen Kosten). • Stützt korrelative Befunde.
Ein Drittel der Lernsituationen war motivierend, aber aufwendig. Using the Experience Sampling • Dieses Ergebnis wäre mit korrelativen Auswertungen übersehen worden.
Method (ESM) to evaluate the motivational characteristics of Die dispositionalen Erwartungen, Werte und Kosten, die Studierende in eine learning situations in a Lernsituation mitbringen, beeinflussen möglicherweise ihr motivationales university lecture Erleben während des Lernens.
Dr. Julia Dietrich 5 / 24
How motivating is a learning situation? Momentary assessments of motivation can be used by researchers and teachers to evaluate how a lesson, learning activity or learning intervention is going. Need for a new type of ESM design
• Typically, ESM studies survey participants at random time points (everyone at a different time point) or at random places. • Therefore, a common question in the interpretation of ESM data is, to what extent do they assess…
How motivating is a learning situation? The situation: Was this an uninteresting situation, everybody would agree?
How motivating is a learning situation? The situation: Was this an uninteresting situation, everybody would agree? The person: Was this person just not interested?
How motivating is a learning situation? The situation: Was this an uninteresting situation, everybody would agree? The person: Was this person just not interested?
Collective (group-level) vs. subjective (individual-level) motivation • We seek to disentangle the collective motivational characteristics of a learning situation, activity or task from individual students’ subjective (possibly diverging) experiences.
Aims 1. To develop an assessment design that allows us to disentangle the collective characteristics of a learning situation (e.g., its interestingness) from subjective perceptions of that same situation (e.g., the subjective situational interest). 2. Identify and calculate coefficients that can be used in the context of motivational interventions.
We present a study design.
We show some advantages of ESM data for interventions. Illustrative data from the Momentary Motivation study (Dietrich, Viljaranta, Moeller, & Kracke, 2017)
Momentary Motivation study: Sample and procedure 155 German university students in a teacher education program • 51% female • Age was M = 21.77 years (SD = 2.91, range 19 to 46 years). • Participants aimed to become subject teachers at secondary schools.
Students were assessed in a weekly lecture on ‘Educational psych’
• Pre-Questionnaire (not included in this study): Beginning of semester (April) • Situational (ESM) assessments: During the semester • Post-Questionnaire (not included in this study): End of semester (July)
Dr. Julia Dietrich 11 / 24
Measures Instruction for the ESM assessment ”The following questions pertain to the lecture contents of the last couple of minutes. … To what extent do the following statements apply to you in the present moment?”
„I understand these contents“
Situational competence belief
Expectancy
„I am interested in these contents“
Situational interest value
Task value
Note. Response format (1 = does not apply to 4 = fully applies).
Dr. Julia Dietrich 12 / 24
Aim 1: An assessment design to disentangle ‚collective‘ vs. ‚subjective‘ motivation
Dr. Julia Dietrich 13 / 24
Aim 1: An assessment design to disentangle ‚collective‘ vs. ‚subjective‘ motivation Design for the ESM assessment
• 10 sessions of 90 minutes, each session dealt with a certain topic • 3 x per lecture 10 questions on momentary motivation (situational expectancies, task values, effort)
Student B
Student A Topic 1: Adolescence Situation 1
Situation 2
Situation 3
Topic 2: Knowledge acquis.
...
Situation 2
...
Situation 1
Situation 3
Topic 1: Adolescence Situation 1
Situation 2
Situation 3
Aim 1: An assessment design to disentangle ‚collective‘ vs. ‚subjective‘ motivation Beeping schedule
• In each session: • Fixed schedules with different groups of students taking the situational assessment at different predetermined time points. • The students of each group evaluate the same learning situation. • Allows to aggregate the group members‘ perceptions of the motivational characteristics of this situation. Sample devided into 3 groups who respond 10, 20, 30, … minutes after start of the lecture at 10 mins
20
30
Group 1
40
50
60
Group 1 Group 2
80
at 90 mins
Group 1 Group 2
Group 3
70
Group 2 Group 3
Group 3
Aim 2: Advantages of this design – Feedback for teachers & researchers
Dr. Julia Dietrich 16 / 24
Some descriptive statistics Multilevel model with situations (Level 1) nested in topics (Level 2) nested in students (Level 3)
• Interest: ICCLevel 2 = .27 (topic level); ICCLevel 3 = .25 (student level) • Competence belief: ICCLevel 2 = .24 (topic level); ICCLevel 3 = .24 (student level) • Correlations between interest and competence belief: • rLevel 1 = .31 (situation level) • rLevel 2 = .67 (topic level) • rLevel 3 = .42 (student level)
The means for the learning situations ranged…
• For interest 2.38 ≤ M ≤ 3.33 (grand mean = 2.95) • For competence belief 2.71 ≤ M ≤ 3.63 (grand mean = 3.23) Note. Response format (1 = does not apply to 4 = fully applies).
Dr. Julia Dietrich 17 / 24
Means Across the Semester (Week 1 to Week 10) 4.00 3.50
Mean
3.00 2.50
Topic „Adolescent development“
2.00 1.50 1.00
1
2
3
Topic „Diagnostics and evaluation“ 4
5
6
7
Week Competence belief
Interest
8
9
10
Aim 2: Advantages of this design Feedback for teachers & researchers Teachers
Researchers
#1 How motivating was a given teaching moment for the class? --> In-situ feedback allows adaptive teaching to increase or maintain motivation.
#3 Which students find which kinds of learning situations very motivating/unmotivating? --> Identify target groups for intervention based on person-environment fit.
#2 How did the average motivation of the class develop over the course of a lesson? --> Momentary motivation data allow in-depth evaluation of particular lessons.
#4 Is the variability of momentary motivation a potential target variable for intervention? --> Is it?
Aim 2: Feedback for teachers
55% of your students were recently interested (beep time was 3:15 pm) 74% of your students felt recently competent (beep time was 3:15 pm) Example: Week 4, beep at 20 mins
#1 How interesting was a given teaching moment for the class? Coefficient Relative frequency of endorsement to motivational statements
Aim 2: Feedback for teachers #2 How did the average interest of the class develop over the course of a lesson? Coefficient Means for the learning situations • interest 2.38 ≤ M ≤ 3.33 (grand mean = 2.95), • for competence beliefs 2.71 ≤ M ≤ 3.63 (grand mean = 3.23). Example. Average Interest in Week 1 (average interestingness of each learning moment) at 10 mins
20
30
3.32
40
50
60
3.43 3.48
80
at 90 mins
3.41 3.35
3.44
70
3.34 3.45
3.44
Aim 2: Feedback for teachers #2 How did the average interest of the class develop over the course of a lesson? Average Motivation in Week 10 (Topic "Diagnostics and Evaluation")
4.00
4.00
3.50
3.50
3.00
3.00 MEAN
MEAN
Average Motivation in Week 1 (Topic "Adolescent development")
2.50 2.00
2.00 Interest
1.50 1.00
2.50
Competence 1
2
3
4
5
6
7
LEARNING SITUATIONS (TIME POINTS)
8
9
Interest What Competence happened 1.00 here 10 during 20 30 40 50 60 70 80 90 class? LEARNING SITUATIONS (TIME POINTS) 1.50
Aim 2: Feedback for researchers 4.00
#3 Which students find which kinds of learning situations very motivating/unmotivating?
3.50 3.00 MEAN
Answering the question: What works for whom under which circumstances?
Average Motivation in Week 10 (Topic "Diagnostics and Evaluation")
2.50 2.00
Coefficients
Between-person variance in within-person parameters = inter-individual differences in intraindividual processes
Interest 1.50 1.00
Competence
10
20
30
40
50
60
70
80
90
LEARNING SITUATIONS (TIME POINTS)
Dana‘s data Lisa‘s data
Example: The trajectory of motivation within Week 10 (intra-individual slope) increases for Dana, but remains flat for Lisa (inter-individual differences in slopes).
Aim 2: Feedback for researchers #4 Is the variability of momentary motivation a potential target variable for intervention? Affect or motivational instability (i.e. high intra-individual variation) could be associated with maladaptive outcomes (e.g., Jahng et al., 2008; Malmberg et al., 2016).
Reducing fluctuations in interest and competence beliefs from one situation or activity to the next -goal of motivational interventions?
Coefficient
Mean square successive difference (MSSD) as a measure of within-person variation (fluctuation from one moment to the next). • interest: variance(MSSD) = .267, p = .043 • competence: variance(MSSD) = .103, p < .001 --> There are inter-individual differences in the amount of fluctuations.
Discussion
Dr. Julia Dietrich 26 / 24
Discussion • Intra-individual methods and their potential to address the motivational heterogeneity of students. • Researchers should look at inter-individual differences in intra-individual processes. • Adaptive teaching interventions possible through the here presented design and momentary smartphone assessments. • Teachers could be given in situ-feedback about how their lesson is going and the motivational characteristics of the learning activities they use. • Validation of individual self-reports by linking individual responses to a collective average of a group’s responses about the same situation. 27 / 24
Discussion • DSEM (dynamic structural equation modeling) applied to the here presented design allows separating within and between person and between time point variance: Yi,s= Y1is + Y2i + Y3s • Y1 = within person deviations separated from • Y2 = stable mean over time, differs for different persons • Y3 = stable mean over persons, differs for different time points
28 / 24
[email protected]
Friedrich-Schiller-University Jena, Germany, Institute of Educational Science
Dr. Julia Dietrich 29 / 24
