Construct Validity Issues in the Measurement of Motivation to Learn
AnneMarie M. Conley and Stuart A. Karabenick University of Michigan, Ann Arbor
Mailing address: 1400 SEB, 610 East University Combined Program in Education and Psychology The University of Michigan Ann Arbor, MI 48109 Phone-734-763-1386 Fax-734-615-2164 Email:
[email protected]
Presented at the biennial meeting of the Society for Research on Adolescence, San Francisco, March 2006. Research reported herein was supported by a grant to the Math and Science Partnership – Motivation Assessment Program (MSP-MAP) from the National Science Foundation (EHR No. 0335369). Views expressed are the authors’ and are not necessarily representative of the funding agency.
Validity and Motivation to Learn
Construct Validity Issues in the Measurement of Motivation to Learn Motivation plays a critical role in student learning and achievement; it is intimately related to the ways students think, feel, and act in schools. Evidence from research on student learning in general (see Pintrich & Schunk, 2002; Pintrich & Maehr, 2004), and mathematics and science in particular (e.g., Fennema, 1989; Schoenfeld, 1992), demonstrates that students’ motivation, affect, strategies, and beliefs about knowledge in these disciplines can influence their learning and performance. Furthermore, research suggests that students’ motivation and related outcomes are sensitive to characteristics of the learning context, including teachers’ instructional practices as well as school and classroom climate (Ames, 1992; Anderman & Maehr, 1999; Eccles & Midgley, 1989). It is important, therefore, for reform efforts to determine how their programs affect student motivation, especially since such changes can precede, or even occur in the absence of, targeted cognitive outcomes. The primary goal of the research reported on here was to develop and make available reliable, valid, and practical tools to assess student motivational beliefs for mathematics and science. These tools are being used with different math and science reform projects to support evidence-based claims about the effects of their interventions, and to explore the role of motivation-related outcomes as mediators and moderators of student achievement in intervention models (Maehr & Karabenick, 2004). There exist a number of different approaches to the study of motivation. For example, consider three different approaches to the question, “What makes students want to learn in school?” One approach is to consider this question in terms of interest, as an individuals’ attraction to, or liking or enjoyment of, a particular task or domain. Another perspective conceptualizes “wanting” in terms of value, a subjective judgment of the degree to which a task
2
Validity and Motivation to Learn or domain can fulfill needs, facilitate reaching goals, or confirm aspects of one’s self-schema. A third approach attends to students’ goals, or their reasons for participating in achievement-related activities. There often is considerable overlap among these constructs, with consistent, moderate correlations among them. Though some have highlighted the importance of doing research that considers these components simultaneously, such research has been limited to date. This paper presents results from a large-scale study of middle and high school students that aimed to address definition and measurement issues by considering multiple constructs deriving from different theoretical traditions simultaneously. Evidence in support of the construct validity of this set of measures to assess motivation to learn is offered using a framework proposed by Messick (1989). Historically, construct validity has been dealt with in different ways, with recent conceptualizations rejecting the traditional three-part (construct, criterion, content) validity approachin favor of a more unified validity theory (Messick, 1989; Pintrich, Wolters, & Baxter, 2000). In Messick’s (1989) unified framework, construct validity is central and other forms of validity are subsumed under it. Messick (1989) proposed a multidimensional framework for thinking about construct validity, and described five kinds of evidence that can be used to support claims of construct validity: content, substantive, structural, external, and generality of meaning. Content-related evidence concerns how well the items reflect the content of the domain. Substantive evidence concerns the relation between data and theory, and the guiding question is whether the data generated by the instrument are consistent with the theory. Structural evidence, on the other hand, concerns the relation between theory and the way the data are reduced: Do the scores obtained reflect the complexities of the theoretical model? External evidence is perhaps the most often considered, and questions include how the instrument relates to other measures of the same
3
Validity and Motivation to Learn construct, and whether the instrument relates to other constructs in theoretically sensible ways. Finally, questions of generality of meaning revolve around how the findings generalize across different populations, contexts, or subject-matter domains. In more recent work, Messick (1995) specified an additional source of evidence, concerned with the intended and unintended consequences of score use. This aspect of construct validity is key when tests are used for assessment or placement decisions, as in the case of performance assessments or standardized tests. Since a complete discussion of all of Messick’s sources of evidence is beyond the scope of this paper, the focus has been narrowed to present substantive, structural, consequential and generality of meaning evidence of the validity of a set of motivation-related measures. The set of motivation-related measures included here draws from the most oftenresearched theoretical frameworks in motivation literature today. These theoretical traditions are characterized by different approaches to the study of the basic questions most research on motivation in education tries to answer: What makes students want to learn in school? What makes students feel competent? How do students’ wants and beliefs in the classroom influence whether and how they approach learning? The research described here draws heavily on expectancy-value theory, achievement goal theory, work on personal and situational interest, and self-efficacy theory. Self-efficacy refers to students’ beliefs that they have the resources and confidence to do the tasks in the classroom (Bandura, 1986; Pintrich & Schunk, 2002). It is important that selfefficacy be calibrated to one’s actual accomplishments (Pintrich & Schunk, 2002). As intervention projects make changes and improve instruction, these reforms may require students to think differently, to do math or science differently, and to engage the material in different ways than is usual in mathematics and science classrooms. Besides beliefs about efficacy and
4
Validity and Motivation to Learn control, task value beliefs are another important motivational component (e.g., Eccles et al., 1998; Pintrich & Schunk, 2002). Longitudinal research by Eccles and her colleagues (e.g., Eccles, et al., 1998; Fredericks, et al., 2002; Jacobs et al., 2002) has shown that student beliefs about the importance and utility of mathematics leads them to enroll in more math courses in the future. In addition, this research has shown that task value beliefs lead to enrollment or choices to take more mathematics courses, but that once enrolled in the actual course, efficacy beliefs are more strongly related to actual performance or achievement. Personal interest refers to an individual's attraction to, or general liking and enjoyment of, a specific activity or domain (Pintrich & Schunk, 2002). Eccles and her colleagues (Eccles, et al., 1998) have shown that personal interest is an important component of motivation and functions similarly to importance and utility value beliefs. In addition, other researchers have shown that high levels of personal interest lead to more cognitive engagement, self-regulation, and achievement (e.g., Koller, et al., 2001; Pintrich & Schunk, 2002). In many mathematics and science reform projects, the goal is to increase student interest and positive attitudes towards mathematics and science domains as well as interest in careers in these areas. It is an important outcome in its own right, as well as a potentially important mediator of achievement (Koller et al., 2001). Another important component of student motivation concerns general achievement goals, or students’ goals for academic learning in classroom contexts. The general distinction between mastery and performance goals contrasts students who are mastery-oriented and focused on learning and understanding and those students who are performance-oriented and focused on doing better than others in terms of grades or other outcomes that invite interpersonal comparisons (Pintrich, 2000a, b). Generally, mastery goals are positive and adaptive and lead to
5
Validity and Motivation to Learn more interest, engagement, and learning. Performance goals, on the other hand, can be adaptive or maladaptive depending on whether students adopt an approach or avoid focus. Performanceavoid goals where students are concerned about looking dumb or trying to avoid getting the lowest scores are clearly maladaptive and are associated with less interest, engagement, and lower levels of performance (Pintrich, 2000a, b). As intervention projects make changes and improve instruction, it is important to understand how these different goals may motivate students to learn and perform in different mathematics and science classrooms.
Method Design Data presented here were obtained through a collaboration between the Motivation Assessment Program at the University of Michigan and a standards-based, data-driven intervention program in the Southwest improve students’ academic performance in mathematics. The collaboration included providing teachers with the knowledge and tools to accurately diagnose students’ deficiencies, assess their progress, adjust the curriculum and pedagogy, and transform the departmental culture to maximize student learning in mathematics. Over the last two years, the partners have collaborated to assess changes in motivation of the more than 14,000 students over the course of the school year. Aggregated analyses of these data were disseminated to teachers and project staff as part of professional development activities that serve as a major component of the intervention, as well as through individual reports to teachers. The professional development activities were designed to change administrative practices in the schools and in the classrooms, effecting a cultural change that creates a sustainable climate of
6
Validity and Motivation to Learn improvement and achievement. The partnership to date has involved five waves of student motivation surveys over two school years, as well as two waves of teacher attitude and belief surveys. Data from the beginning and end of the first year of student surveys are presented here. Students were administered questionnaires by research assistants in their regular math classrooms four weeks after the start of the school year and again approximately four weeks before the end of the school year. All students in class on the day of administration participated. Students were told that the purpose of the confidential survey was to elicit their thoughts and feelings about the subject of math and their own math class. Students were guided through a sample item and then completed a 110- question survey during their math period. Items were read aloud to the middle school students; high school students worked through the survey independently after receiving instructions from trained research assistants. The survey took approximately 30 min. to complete. The teacher was present in the room while the survey was being completed, but remained seated and unobtrusive, unable to view any of the survey responses. Participants Analyses presented here are based on 8,429 students (49% female) from 487 classrooms in 14 ethnically diverse, working class public middle and high schools in California (72% Latino/a, 16% Vietnamese, 6% Caucasian, 6% Other - primarily SE Asian). Between 60 and 75% of the students in these schools were eligible to receive free or reduced lunch. Two of the four districts have been characterized as high-need districts by the state and there is a sizable population of English Learners.
7
Validity and Motivation to Learn Measures The student motivation survey included measures of self-efficacy for learning math and solving math problems, task value for, and students’ personal achievement goals. Items were rated on a 5-point Likert scale (1 = not at all true; 3 = somewhat true; 5 = very true), and all questions were worded to have students focus on the domain of mathematics. Task Value was measured with 18 items, which included four components adapted from previous work. Interest (6 items, ! = .95) referred to students’ attraction to, liking for, and enjoyment of math. (e.g., “I find math very interesting”). Utility (6 items, ! = .87) was concerned with students’ beliefs about the usefulness of math as an area of study (e.g., “Math is useful to me for things I do outside of school”). While utility value focused on the importance of math as a means to an end, attainment value focused on the value of math as part of a student’s identity. Attainment value (6 items, ! = .87) referred to students’ judgments about the importance of math for their sense of who they are (e.g., “Thinking mathematically is an important part of who I am”). Cost value (2 items, ! = .81) tapped students’ judgments about the amount of effort required to be successful in math (e.g., “Success in math requires that I give up other activities I enjoy”). Efficacy (8 items, ! = .88) items assessed students’ judgments about their ability and confidence to perform adequately in math (e.g., “How sure are you that you can do even the most difficult math work”). Achievement goals (three 5-item scales items, !s = .87, .84, .79) referred to students’ purposes when approaching, engaging in, and responding to math instruction. Mastery goals focused on learning and understanding (e.g., “My goal in math is to learn as much as I can”), performance-approach goals
8
Validity and Motivation to Learn focused on demonstrating ability and outperforming others (e.g., “My goal in math is to look smarter than other students”), and performance-avoid goals focused on not looking dumb (e.g., “My goal in math is to avoid looking like I can’t do my work”). Substantive Evidence Substantive evidence is concerned with the internal relations among the items in an instrument. The guiding question is whether the data generated by the instrument are consistent with the theory of the construct. The measures of task value and achievement goals included in our assessment would show substantive evidence of construct validity, according to Messick (1989), if the number and type of scores generated was consistent with the theories from which the items were developed. In the case of task value, four components are predicted: interest, utility, attainment, and cost. Previous studies have sometimes had difficulty finding the predicted distinctions between utility value and attainment value. Results from exploratory factor analyses of the task value items are presented in tables 1 (beginning of school year) and 2 (end of school year). Structural Evidence This component of construct validity asks whether the scoring of the instrument reflects the complexities of the theory. A single total score indicates a unitary construct, while a combination of composite scores and subscores indicates a hierarchical construct. Subsumed under the structural component are issues related to scale reliability. With older validity theories, reliability was separate from validity. This made it possible to have scales that were valid, but not reliable, or scales that were reliable, but not valid. With Messick’s (1989) unified approach to construct validity, issues of reliability are factored into judgments of validity. Reliabilities for the achievement goal scales for this
9
Validity and Motivation to Learn sample are in line with previous work, which usually shows high reliability as assessed by indices of internal consistency (! " .85) for the mastery and performance approach scales, but lower reliability for the avoidance scales. Table 3 shows reliabilities for both waves, presented separately for the middle and high school students. Consequential Evidence A critical component of construct validity for our project concerns the intended and unintended consequences of score use. Messick’s discussions dealt with performance assessments or standardized tests and the use of those scores for assessment or placement decisions. Motivation-related data are not typically associated with these same kinds of immediate consequences for students, however the nature of our partnership is not typical. Much of our work is focused on disseminating data to teachers through professional development that targets change in teacher practice to support adaptive motivation for students. Figure 1 presents a sample report generated for a participating school. Such reports are used in professional development workshops to help teachers and schools find areas of strength and weakness from a motivational perspective. The decision to present these results in their full complexity was made in response to a general tendency to oversimplify motivation in the classroom. For example, teachers would report that their students were simply “unmotivated.” Therefore, one aim of our project was to show that there were different ways for students to be motivated (and unmotivated), and that these different ways required different interventions from teachers when problems arose. Results were presented to teachers separated by course, because teachers wanted to address motivational concerns in course-alike teams. The quality of the motivational
10
Validity and Motivation to Learn problems with Algebra 1 students was quite different from the quality of the problems with Pre Calculus students. In response, we structured the reporting of data to support these conversations. The intended consequences of the use of these scores included changes in the content of professional development and changes in the schools’ action plans. Along with the graphs, we provided interpretations designed to facilitate discussion. This is an excerpt describing Algebra 1A students in middle school “A”: •
Algebra 1A students started the year with the lowest scores on interest, mastery, and efficacy (e.g., they saw math as less interesting than other math students, they were less likely to focus on understanding, and were the least confident in their math ability). However, they saw math as just as useful as other students in the school, and had similar levels of focus on competition. o Change –Algebra 1A students had a more adaptive pattern of change than other students at this school; the drops were generally smaller than for students in other courses. They saw math as less useful and were less focused on learning but slightly more confident in their math ability. o Goals for next year – Help students see how math is useful, and more importantly, use TARGET TIpS to help focus students on learning and developing (rather than just demonstrating) ability. With 14 different schools as collaborators, we have seen variation in the degree to
which the motivational data have had consequences in terms of teacher practices, and student outcomes. In some schools, detailed yearly action plans have been revised to include a focus on motivation. In one school the drop in student interest presented the biggest concern for teachers, and the math department has made supporting student interest a major focus. It is more difficult to report at this point on the unintended consequences of score use, but such questions will be important to examine as we continue to examine the construct validity of our set of motivation measures.
11
Validity and Motivation to Learn Evidence of Generality of Meaning This component of construct validity concerns how the scores on an instrument generalize to other populations and contexts. Of particular interest for this set of motivation measures are characteristics of the sample. This study included an ethnically diverse sample of 6th through 12th students, and we found the decline in motivation across middle and high school reported in other studies (e.g., Anderman, Maehr, & Midgley, 1999). Figures 2 and 3 show beginning and end of school year motivation profiles, separated for each grade level. Students showed expected drops in motivation over the course of the school year. They became less interested, saw math as less useful, and felt less confident in their ability to understand math. In addition, they reported lower levels of achievement goals, with lower means on all three goals. While a decreased focus on mastery goals of learning and understanding is problematic, the associated decrease in a focus on competition should be considered an adaptive change. The overall decrease in motivation across the school year played a smaller role in professional development than the variability we found across schools and between courses. Looking at variability across courses formed the basis for much of the dialogue during professional development activities. For example, sixth graders were particularly disadvantaged over the school year, with the greatest drop-offs over the year. They were less interested, considered math less useful, and were less confident in their math abilities. A positive change was the decreased focus on competition and not looking incompetent, but this was accompanied as well by less of a focus on learning and understanding. Targeted professional development with sixth grade teachers has focused
12
Validity and Motivation to Learn on supporting mastery goals, interest, and value over the school year. Other questions of generality of meaning concern possible gender or ethnic differences, or even domainspecific differences, in the structure of these constructs. As data from different populations and science intervention projects becomes available, more evidence of the generality of these measures will be investigated. Discussion Exploratory factor analysis, reliability analysis, and figures showing grade-level differences are offered here as substantive, structural, and generality of meaning evidence in support of the construct validity of this set of measures of motivation to learn. Further, a discussion of the ways in which these data have been reported and used serves as consequential evidence. Though these sources of evidence have been separated for clarity of discussion, it is important to remember that there exists considerable overlap, and that these aspects of construct validity are part of a unified validity theory that does not rely on nor require any one form of evidence.
13
Validity and Motivation to Learn References Ames, C. (1992). Classrooms: Goals, structures, and student motivation. Journal of Educational Psychology, 84, 261-271. Anderman, E. M., M. L. Maehr, et al. (1999). Declining motivation after the transition to middle school: Schools can make a difference. Journal of Research and Development in Education, 32(3), 131-147. Eccles, J. S. and C. Midgley (1989). Stage-environment fit: Developmentally appropriate classrooms for young adolescents. In C. Ames and R. Ames (Eds.), Research on motivation in education (pp. 139-186). San Diego, CA: Academic Press. Eccles, J., Wigfield, A., Harold, R.D., & Blumenfeld, P. (1993). Age and gender differences in children’s self- and task perceptions during elementary school. Child Development, 64, 830-847. Eccles, J., Wigfield, A., & Schiefele, U. (1998). Motivation to succeed. In W. Damon (Series Ed.) & N. Eisenberg (Vol. Ed.), Handbook of child psychology: Vol. 3. Social, emotional, and personality development (5th ed., pp. 1017-1095). New York: Wiley. Fennema, E. (1989). The study of affect and mathematics: A proposed generic model for research. In D. McLeod & V. Adams, (Eds.), Affect and mathematical problem solving: A new perspective (pp. 205-219). New York: Springer. Koller, O., Baumert, J., & Schnabel, K. (2001). Does interest matter? The relationship between academic interest and achievement in mathematics. Journal for Research in Mathematics Education, 32, 448-470. Maehr, M. L., & Karabenick, S. A. (April, 2004). Tools for the evaluation of non-cognitive outcomes of math and science instruction. Paper presented as part of a symposium entitled:
14
Validity and Motivation to Learn Supporting evidence-based claims for impacts of science and mathematics reform on student outcomes, at the annual meeting of the American Educational Research Association, San Diego, CA. Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational Measurement (3rd ed., pp. 13104). New York: Macmillan. Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons' responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9), 741-749. Pintrich, P.R. (2000a). Multiple goals, multiple pathways: The role of goal orientation in learning and achievement. Journal of Educational Psychology. 92, 544-555. Pintrich, P.R. (2000b). The role of goal orientation in self-regulated learning. In M. Boekaerts, P.R. Pintrich, & M. Zeidner, (Eds.). Handbook of self-regulation (pp. 451-502). San Diego, CA: Academic Press. Pintrich, P.R., & Schunk, D.H. (2002). Motivation in education: Theory, research, and applications (2nd ed.). Upper Saddle River, NJ: Merrill Prentice Hall. Pintrich, P.R., & Maehr, M.L. (Eds.). (2004). Advances in motivation and achievement: Motivating students, improving schools (Vol. 13). Oxford, England: JAI, An Imprint of Elsevier Science. Pintrich, P. R., Wolters, C. A., & Baxter, G. P. (2000). Assessing metacognition and selfregulated learning. In G. Schraw & J. C. Impara (Eds.), Issues in the measurement of metacognition (pp. 43-97). Lincoln, NE: Buros Institute of Mental Measurements. Schoenfeld, A.H. (1992). Learning to think mathematically: Problem solving, metacognition, an sense-making in mathematics. In D. Grouws, (Ed.), Handbook of research on mathematics
15
Validity and Motivation to Learn teaching and learning (pp. 334-370). New York: Macmillan.
16
Validity and Motivation to Learn Table 1 Factor Loadings for Task Value Measures at Beginning of School Year (N = 8,429)
1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20.
I enjoy doing math. I like math. I enjoy the subject of math. How much do you like doing math? Math is exciting to me. I am fascinated by math. Math will be useful for me later in life. Math concepts are valuable because they will help me in the future. How useful is learning math for what you want to do after you graduate and go to work? In general, how useful is what you learn in math? Being good at math will be important when I get a job or go to college. Compared to most of your other school subjects, how useful is what you learn in math? I have to give up a lot to do well in math. Success in math requires that I give up other activities I enjoy. It is important for me to be someone who is good at solving problems that involve math. Being someone who is good at math is important to me. Being good at math is an important part of who I am. It is important to me to be a person who reasons mathematically. I feel that, to me, being good at solving problems which involve math or reasoning mathematically is Thinking mathematically is an important part of who I am.
Note: Factor loadings under .20 have been omitted.
1 0.95 0.93 0.88 0.85 0.83 0.75
2
3
4
0.94 0.83 0.71 0.60 0.52 0.46 0.79 0.77 0.79 0.79 0.74 0.63 0.61 0.55
17
Validity and Motivation to Learn Table 2 Factor Loadings for Task Value Measures at End of School Year (N = 8,429)
1. 2. 3. 4. 5. 6. 7. 8.
I enjoy doing math. I like math. I enjoy the subject of math. How much do you like doing math? Math is exciting to me. I am fascinated by math. Math will be useful for me later in life. Math concepts are valuable because they will help me in the future. 9. How useful is learning math for what you want to do after you graduate and go to work? 10. In general, how useful is what you learn in math? 11. Being good at math will be important when I get a job or go to college. 12. Compared to most of your other school subjects, how useful is what you learn in math? 13. I have to give up a lot to do well in math. 14. Success in math requires that I give up other activities I enjoy. 15. It is important for me to be someone who is good at solving problems that involve math. 16. Being someone who is good at math is important to me. 17. Being good at math is an important part of who I am. 18. It is important to me to be a person who reasons mathematically. 19. I feel that, to me, being good at solving problems which involve math or reasoning mathematically is 20. Thinking mathematically is an important part of who I am. Note: Factor loadings under .20 have been omitted.
1 0.96 0.94 0.86 0.86 0.83 0.76
2
3
4
0.96 0.88 0.75 0.65 0.61 0.51 0.83 0.82 0.81 0.80 0.80 0.73 0.65 0.63
18
Validity and Motivation to Learn Table 3 Reliabilities for Efficacy, Task Value, and Achievement Goal Measures for Beginning of School Year (N = 8,429)
Interest Value Utility Value Attainment Value Cost Value Personal Mastery Approach Goals Personal Performance Approach Goals Personal Performance Avoid Goals Efficacy
Middle School
High School
Combined
0.95
0.96
0.96
0.87
0.84
0.87
0.87
0.86
0.87
0.79
0.71
0.75
0.86
0.87
0.87
0.84
0.84
0.84
0.80
0.78
0.79
0.89
0.87
0.88
19
Validity and Motivation to Learn Table 4 Reliabilities for Efficacy, Task Value, and Achievement Goal Measures for End of School Year (N = 8,429)
Interest Value Utility Value Attainment Value Cost Value Personal Mastery Approach Goals Personal Performance Approach Goals Personal Performance Avoid Goals Efficacy
Middle School
High School
Combined
0.95
0.96
0.95
0.90
0.88
0.89
0.90
0.88
0.89
0.83
0.78
0.81
0.88
0.88
0.88
0.86
0.86
0.86
0.84
0.82
0.83
0.91
0.91
0.91
20
Validity and Motivation to Learn Figure 1 Sample School Report
21
Validity and Motivation to Learn Figure 2 Cross-sections of Beginning-of-Year Motivation Profiles for Middle and High School Students (N = 8,429)
22
Validity and Motivation to Learn Figure 3 Cross-sections of End-of-Year Motivation Profiles for Middle and High School Students (N = 8,429)
23
