Measurement of STEM Dispositions in Elementary School Students Gerald Knezek Rhonda Christensen Tandra Tyler-Wood Sita Periathiruvadi Curby Alexander Garry Mayes University of North Texas, USA. Charlotte Owens Dale Magoun University of Louisiana Monroe, U.S.A. Contact:
[email protected] Abstract: This paper focuses on the development and refinement of a new instrument for measuring Science, Technology, Engineering, and Math (STEM) dispositions in elementary school students. Reliability and validity indices are presented, as well as a cross-validation study comparing the new instrument to established middle school measures. Four of the five proposed scales are found to be reliable and have reasonable construct validity, while the fifth is borderline and in need of improvement. This instrument will add to the repertoire of articulated instruments under development to measure the same constructs in student, teacher, and STEM career professionals, eventually enabling better-focused teacher preparation.
Introduction Fab@School is an NSF ITEST (National Science Foundation Innovative Technology Experiences Students and Teachers) project (NSF ITEST #1030865) funded for three years beginning in 2010. Leading university partners are the University of Virginia, Cornell University, and the University of North Texas. The Classroom Fab@School coalition was established to prepare students for the workforce revolution and encourage them to pursue STEMrelated careers by developing fabrication laboratories (Fab@School) for the elementary classroom. Fab@School makes digital fabrication in the elementary and middle-school grades scalable and will allow students to learn skills and concepts such as 3D visualization that are equally applicable to larger industrial systems. Fab@School is being used to enhance technology, mathematics, and engineering (STEM) instruction while preparing students for the STEM workforce. The goals of the Fab@School project are: • •
• •
Develop a comprehensive system for introducing digital fabrication in the elementary grades that integrates hardware, software, a curriculum, and a collaborative space, Increase preservice elementary teachers' competence and interest in teaching STEM content, and specifically the mathematics that underpins engineering proficiency through introduction of digital fabrication in preservice teacher education courses, Increase in-service teachers' competence and interest in teaching STEM content through professional development workshops that introduce digital fabrication in the context of mathematics teaching, and Increase elementary children's competence in mathematics while simultaneously increasing their interest in engineering and STEM careers by engaging and supporting preservice teachers and continuing that support through their first year of teaching.
- 1052 -
The Need for STEM Measures Nationwide in the USA efforts are being made to improve STEM (science, technology, engineering and mathematics) education and make it a national priority to strengthen America’s position in discovery and innovation amidst global challenge (The White House, 2009). Unfortunately, many students at the post-secondary level leave STEM majors because of lack of preparation to meet the demanding coursework in college (Stramel, 2010). Elementary students can gain more knowledge and interest in science when they engage in hands-on activities similar to everyday activities of professionals in the STEM fields. Several researchers have found support for inquiry-based science instruction to not only improve science literacy and achievement, but also develop positive attitudes of elementary students towards science (Minogue, Madden, Bedward, Wiebe & Carter, 2010). According to many sources, STEM career intervention and enrichment plans should be initiated well before the high school years. As education and popular perception of technology and engineering standards evolve there is also an increased awareness of the need for STEM literacy within society. STEM literacy fosters intelligent participation in public socio-scientific and ethical decisions that direct the future of engineering and technology (Gorham, 2002; Stiller, De Miranda, & Whaley, 2007). Research studies have determined that factors influencing students’ attitudes towards science include influences of teachers, parents and peers (George, 2006). However, little is known about how positive attitudes towards STEM are nurtured at the elementary school level, and whether positive attitudes persist over time and foster future careers in STEM. Accurate measurement of STEM activities and dispositions at the elementary school level is a fundamental requirement for answering these and other related research questions. The instrumentation under development and refinement for the Fab@School project is a first step in the accurate assessment of progress toward project goals. Instrument development and validation is the focus of this paper.
Instrumentation: New Development and Cross-Validation Scales Prior work has been completed by the authors in the measurement of STEM dispositions at the middle school level (Tyler-Wood, Knezek & Christensen, 2010) This was in conjunction with another NSF ITEST project called Middle Schoolers Out to Save the World (MSOSW), where teachers guided their students in home monitoring of standby power consumption in electronic devices, then aggregated data at the classroom level for making projections on how to reduce greenhouse gas emissions. Instruments that were used for middle school students in this three-year project are described in the sections that follow.
STEM dispositions and STEM career attitudes. The STEM Semantics survey (Tyler-Wood, Knezek & Christensen, 2010) is a 25-item instrument that measures interest in each STEM subject as well as interest in STEM careers more generally. The internal consistency ratings for the five subscales ranged from 0.88 to 0.93, which can be considered very good (DeVellis, 1991). Each of the five scales has semantic adjective pairs (interesting: boring; appealing: unappealing; exciting: unexciting and so forth) as descriptors of STEM dispositions and career attitudes. A copy of this instrument is provided in the Appendix.
Computer attitudes and learner dispositions. Several subscales from the Computer Attitude questionnaire (CAQ) (Knezek, Christensen, Miyshita, Ropp, 2000; Mills, Wakefield, Najmi, Surface, Christensen & Knezek, 2011) were used for the middle school project. Among these were Computer Enjoyment, Computer Importance, Computer Learning, Computer Comfort, Motivation/Persistence, Creative Tendencies, and Attitudes Toward School. The subscales contained Likert-type items with ratings ranging from strongly disagree (1) to strongly agree (5). The
- 1053 -
reliability estimates for the subscales were most recently analyzed by Mills et al., (2011) and found to be in the range 0.73 to 0.85 in the MSOSW project. These fall in the range of respectable to very good according to the guidelines provided by DeVellis (1991).
New Instrument for Grades 4 and 5. Baseline project data were gathered from 89 fourth and fifth graders during the first year of the Fab@School Project in 2010-2011. These data were sufficiently reliable for providing formative feedback to teachers and staff during the curriculum refinement phase of the project. However, Fab@School teachers and project support staff determined the majority of the items validated for middle school students (grades 6-8) were too difficult for 4th and 5th grade students. Therefore work began on the development and refinement of a new instrument based on items from the National Assessment of Educational Progress (NAEP, 2011) and The International Mathematics and Science Study (TIMSS, 2007) that had been developed for the elementary school level. Thirty-two items in five sections were selected and completed by 38 of the project’s 4th and 5th graders in fall 2011.
Cross-Validation Phase The research team for the Fab@School project successfully pilot-tested and refined middle school STEM disposition instruments with university-level teacher preparation candidates in the past (Knezek, Christensen & Tyler-Wood, 2011). A similar process was used for the new instruments targeted at the elementary school level. Specifically, three classes of an introductory technology integration course that completed Fab@School activities as part of their coursework (and one comparison class that did not) responded to items from the new elementary school questionnaire along with the standard pre-post assessment. Specifically, subjects for the cross-validation stage were preservice teacher preparation students from four different sections of a Computers in the Classroom course. These university students were typically in their second or third year of a four-year undergraduate degree program. As shown in Table 1, 72 teacher preparation candidates completed the 32 items targeted for elementary school students, as well as the previously-validated STEM Semantic Survey. Table 1. Internal Consistency Reliabilities for 72 Preservice Educators on Ten STEM Disposition Scales. Scale Cronbach’s Alpha Cronbach’s Alpha Cronbach’s Alpha Treatment Classes w/ Comparison Univ. + Elem. (n=54) Class (n=72) (n= 110) Section A – Engagement in .78 .72 .55 (only 6 items) Hands-on Science Section B – Classroom .85 .85 .84 Experiences in Science Section C – Science Self Efficacy/ .64 .65 .66 Persistence Section D – Engagement in .77 .71 .72 (only 8 items) Mathematics (one group collaboration item) Section E – Mathematics Self.92 .90 .88 efficacy TIMSS Perception of Math .94 .94 Not completed STEM Semantic Science .92 .90 “ STEM Semantic Math .86 .88 “ STEM Semantic Technology .72 .72 “ STEM Semantic Engineering .89 .89 “ STEM Semantic - STEM Career .89 .92 “
- 1054 -
Number of items 7 2 8 9 6 8 5 5 5 5 5
Cronbach’s Alpha was calculated for 72 preservice teachers combined with 38 fourth and fifth graders to assess the internal consistency of the new items (Sections A – E) grouped by the Fab@School external evaluation team together with project personnel. This procedure was used to determine the extent to which the set of items in each of the five sections functioned together as a unidimensional scale. As shown in Table 1, Cronbach’s Alpha for Sections A – E ranged from .64 to .85 for preservice teachers, and .55 to .84 for the elementary student data and preservice teachers combined (n=110). One is currently “unacceptable” (Elem. Section A = .55). The others fall in the range of “undesirable” to “very good” according to the guidelines by DeVellis (1991) provided in Table 2. Table 2. DeVellis Reliability Guidelines. Below .60 Between .60 and .65 Between .65 and .70 Between .70 and .80 Between .80 and .90 Much above .90 (DeVellis, 1991, p.85).
Unacceptable Undesirable Minimally acceptable Respectable Very good Consider shortening the scale
The same procedure applied to the new collection of items was also applied to the five subscales (25 items) of the previously validated STEM Semantic Survey. The TIMSS Perception of Math Scale (TIMSS, 2007) was also subjected to this procedure. Cronbach’s Alpha was computed in order to re-verify the proper functioning of these items as a scale for this new set of data. As shown in the lower half of Table 1, these values ranged from .72 to .92. All were in the range of “respectable” to “very good” according to guidelines by DeVellis (1991) shown in Table 2. Construct Validity. Exploratory factor analysis (principal components, varimax rotation) was conducted on each of the sections of the new instrument to examine whether more than one construct was likely to exist in any one section. Data were from 3 classes (n = 54) of preservice educators. Preliminary findings were: •
•
•
Section A (7 items) likely contains 1-3 constructs and forms a reasonable unidimensional scale (Alpha = .72 - .78) as described in Table 1. The seven items in this section are: 1. How often have you done activities or projects in science? 2. How often have you worked with other students on a science activity or project? 3. How often have you written a report on your science activities or projects? 4. How often have you presented what you learned about science to your class? 5. How often have you used a computer for science? 6. How often have you used other digital devices other than a computer for science (for example fabricating printer, meters or science kits)? 7. How often have you talked about measurements or results from your science activities or projects? Section B (2 items) likely contains 1 construct and forms a reasonable unidimensional scale (Alpha = .85) as described in Table 1. The two items in this section are: 1. How often have you done activities or projects to learn about electricity (for examples, batteries, & light)? 2. How often have you done activities or projects using simple machines (for example, pulleys & levers)? Section C (8 items) likely contains 2 constructs, the first composed of times 2, 1, 3, 8 (Science Affinity) and the second composed of items 6, 5, 7, 4 (Motivation / Persistence). This section, if taken as a whole, forms a weak (Alpha = .64 - .65) unidimensional scale. A brief description of each item and its factor loading matrix is presented in Table 3. The full wording of the eight items in this section are: 1. How often do you understand what the teacher talks about in science class? 2. How often do you feel you can do a good job on your science assignments? 3. How often do you feel science is one of your favorite subjects?
- 1055 -
4. 5. 6. 7. 8.
When I am faced with a difficult problem, I keep working until I find the answer. I enjoy working on a difficult problem. Before I start to solve a problem, I make a plan. I like to work on problems which I can use in my everyday life. Scientists make a meaningful difference in the world.
Table 3. Two-Factor Structure for Section C of Elementary Student STEM Survey. Rotated Component Matrix 1. Science Affinity SectC_2. Do good job on science assignment SectC_1. Understand what teacher talks about SectC_3. Science is one of favorite subjects SectC_8. Scientists make a difference in the world SectC_6. Make a plan for a difficult problem SectC_5. Enjoy working on difficult problem SectC_7. Enjoy working on problems to use in everyday life SectC_4. Keep working on difficult problem until find answer
.863 .810 .787 .488 .129 .312
2. Motivation/ Persistence .141 .192 .742 .687 .686 .577
•
Section D (9 items) likely contains 1-3 constructs and forms a reasonable unidimensional scale (Alpha = .71 - .77) as described in Table 1. The nine items (TIMSS, 2007) in this section are: 1. I practice adding, subtracting, multiplying and dividing without using a calculator……... 2. I work on fractions and decimals………. 3. I measure things in the classroom and around the school……… 4. I make tables, charts, or graphs…….. 5. I learn about shapes such as circles, triangles, rectangles, and cubes…... 6. I memorize how to work problems…… 7. I work with other students in small groups…… 8. I work mathematics problems on my own……. 9. I use a computer………
•
The STEM Semantic Survey contains 5 previously validated constructs. These are listed in Table 1. The entire survey is listed in the Appendix.
Concurrent Validity. Correlation coefficients were computed between the established scales and the new scales to determine concurrent validity. As shown in Table 4, the Section C scale score (Science Self-Efficacy / Persistence) is strongly correlated (r = -.51, p < .0005) with Semantic Perception of Science, while Section E (Mathematics SelfEfficacy) is strongly correlated (r = .66, p < .0005) with Semantic Perception of Math. In addition, Section D (Engagement in Mathematics) is moderately correlated with Semantic Perception of Math (r = .36, p < .002) and Semantic Perception of Engineering (r = -.28, p < .02). There were no significant correlations found between Section A (Engagement in Hands-on Science), nor Section B (Classroom Experiences in Science) and the STEM Semantic Perception scales. Note that the negative correlations found in Table 4 are an artifact of the high to low ordering of the rating categories on several of the new scales, rather than the low to high ordering common on previously developed instruments. All significant associations are positive, in that more positive agreement with interest in STEM on one scale corresponds more positive Semantic Perception of STEM on the corresponding scale.
- 1056 -
Table 4. Correlations between STEM Elementary Scales and STEM Semantics. SCIAVG SecATot
SecBTot
SecCTot
SecDTot
SecETot
Pearson Correlation Sig. (2-tailed) N Pearson Correlation Sig. (2-tailed) N Pearson Correlation Sig. (2-tailed) N Pearson Correlation Sig. (2-tailed) N Pearson Correlation Sig. (2-tailed) N
MathAvg
-.119 .323 71 .111 .357 71 -.511** .000 71 -.008 .948 70 .205 .084 72
.065 .589 71 .043 .724 71 -.199 .097 71 -.363** .002 70 .660** .000 72
EngAvg .000 .998 71 .096 .428 71 -.361** .002 71 -.280* .019 70 .132 .268 72
TechAvg .106 .377 71 -.053 .660 71 .068 .574 71 -.121 .317 70 -.123 .302 72
StemAvg -.156 .194 71 -.029 .811 71 -.293* .013 71 -.411** .000 70 .407** .000 72
Discussion Most scales produced from the STEM Elementary student dispositions instrument were found to be at least minimally acceptable. Plans for future refinements include: 1. Dividing the Sections into construct-based scales when appropriate. For example, Section C appears to contain two constructs in one section. It is likely that other sections can be subdivided to produce more reliable measures. 2. Making the item rating scales all have the same direction as greater agreement or more positive (Strongly Disagree = 1 and Strongly Agree = 5), rather than having some blocks of items with 1 = Strongly Agree. 3. Using a standard Likert rating category format with at least 4 choices, in order to increase prospects for variation in responses. Currently many scales have just three choices, such as: a) “Every or almost every lesson,” “Some lessons,” or “Never”; or b) “Always or almost always,” “Sometimes,” or “Never.”
Summary and Conclusions The primary purpose of this paper is to examine the measurement characteristics of a new STEM dispositions instrument designed for use at the 4th and 5th grade level. Major findings are that four of the five proposed scales are reliable and have reasonable construct validity, while the fifth is borderline (r =.55 vs. r = .60 considered minimally acceptable), but could still be useful. This work is being undertaken in an attempt to have an instrument in place in time for pre-post assessment of young scientists in the Fab@School project, in order to determine if the activities led by teachers have resulted in measurable disposition changes. Preliminary indications are that a useful instrument has emerged. This instrument will add to the repertoire of articulated instruments under development to measure the same constructs in student, teacher, and STEM career professionals. Comparisons of dispositions of elementary and middle school students to those of preservice teachers and STEM education professionals have already led the authors of this paper to establish new disposition targets for preservice educators, because they will be the conduit through which our society hopes that interest in STEM careers for our future adult workforce will flow. Instruments such as the one introduced in this paper could help teacher educators identify or develop a future teacher workforce capable of fostering increased interest in STEM careers.
- 1057 -
References DeVellis, R.F. (1991). Scale development. Newbury Park, NJ: Sage Publications. George, R. (2006). A Cross- domain analysis of change in students’ attitudes toward science and attitudes about utility of science. International Journal of Science Education, 28 (6), 571-589. Retrieved from http://www.tandf.co.uk/journals/titles/09500693.asp. Gorham, D. (2002). Engineering and standards for technological literacy. The Technology Teacher, 61(7), 29. Institute for the Integration of Technology into Teaching and Learning. (2010). Middle schoolers out to save the world. Retrieved from http://iittl.unt.edu/IITTL/itest/msosw_web/. Knezek, G., Christensen, R., Miyashita, K., & Ropp, M. (2000). Instruments for assessing educator progress in technology integration. Denton, TX: Institute for the Integration of Technology into Teaching and Learning (IITTL). Knezek, G., Christensen, R., & Tyler-Wood, T. (2011). Contrasting perceptions of STEM content and careers. Contemporary Issues in Technology and Teacher Education, 11(1), 92-117. Mills, L., Wakefield, J., Najmi, A., Surface, D., Christensen, R. & Knezek, G. (2011). Validating the computer attitude Questionnaire NSF ITEST (CAQ N/I). In M. Koehler & P. Mishra (Eds.), Proceedings of Society for Information Technology & Teacher Education International Conference 2011 (pp. 1572-1579). Chesapeake, VA: AACE. Minogue, J., Madden, L., Bedward, J., Wiebe, E., & Carter, M. (2010). The Cross-Case Analyses of Elementary Students’ Engagement in the Strands of Science Proficiency. Journal of Science Teacher Education, 21(5), 559-587. doi:10.1007/s10972-010-9195-y. NAEP (2011). National assessment of educational progress. Retrieved from http://nces.ed.gov/nationsreportcard/ Stramel, J. (2010). A naturalistic inquiry into the attitudes towards mathematics and math self-efficacy beliefs of middle school students. Retrieved from http://krex.kstate.edu/dspace/bitstream/2097/4631/1/JanetStramel2010.pdf. Stiller, T., De Miranda, M., & Whaley, D. (2007). Engineering education partnership. The International Journal of Engineering Education, 23(1), 58. TIMSS (2007). Trends in international math and science study. Retrieved from http://timss.bc.edu/TIMSS2007/context.html. Tyler-Wood, T., Knezek, G., & Christensen, R. (2010) Instruments for assessing interest in STEM content and careers. Journal of Technology and Teacher Education, 18(2), 341-363. The White House. (2009). Educate to innovate campaign for excellence in science, technology, engineering & math (stem) education. Retrieved from http://www.whitehouse.gov/the-press-office/president-obama-launcheseducate-innovate-campaign-excellence-science-technology-en. Acknowledgement This research was supported in part by U.S. National Science Foundation Innovative Technologies Grant #1030865.
- 1058 -
Appendix: STEM Semantics Survey
Gender: M / F
This five-part questionnaire is designed to assess your perceptions of scientific disciplines. It should require about 5 minutes of your time. Usually it is best to respond with your first impression, without giving a question much thought. Your answers will remain confidential.
Instructions: Choose one circle between each adjective pair to indicate how you feel about the object. To me, SCIENCE is:
! "
To me, MATH is:
! "
! "
To me, TECHNOLOGY is:
! "
To me, a CAREER in science, technology, engineering, or mathematics (is):
! "
#$%& '
(
) *
+ ,- . /
- 1059 -
"0 ((1
