Bayesian Networks to Model Science Inquiry Skills in ...

Bayesian Networks to Model Science Inquiry Skills in NAEP ICTs Johnny Lin, Margarita Olivera-Aguilar, Yue Jia Educational Testing Service Paper accepted for the 2015 National Council of Measurement in Education Conference In 2004, the National Assessment Governing Board (NAGB) proposed a new science framework (Champagne et al., 2004) that was eventually implemented in 2009 (US. Department of Education) which proposed to bring together advances in cognitive science and measurement theory to measure abilities such as scientific inquiry skills that were difficult to measure previously. One of these specifications was the development and administration of the 2009 National Assessment of Educational Progress (NAEP) science interactive computer tasks (ICTs). The NAEP 2009 Science ICTs contain 20 or 40 minute tasks designed to assess how well students can perform scientific investigations, draw valid conclusions, and explain their results (NAEP tech report). These skills can be assessed through POE item sets (NAEP Framework 2009, p.124), notably 

Predict. Students provide a prediction for what might happen in a real-world science situation



Observe. Students conduct an investigation and observe what happens



Explain. Students explain what they have observed by interpreting data or drawing conclusions

Complex observables such as POE item sets administered through interactive computer tasks pose particular challenges for psychometric modeling for Traditional Rasch-based unidimensional item response theory (IRT) models may be limited to analyze assessments in which a sequential framework is hypothesized such as the one in NAEP Science. If researchers are interested in making claims about scientific inquiry skills through a Predict, Observe and Explain sequential framework an alternative psychometric model is needed. One proposed alternative is the Bayesian networks (see Mislevy et al. 1999 for its application to educational assessment). Bayesian networks combine concepts of probability theory with graphical modeling to model joint relationships among observable nodes (see Figure 1). For discrete random variables, the parameters of interest are conditional probabilities tables (CPTs). The joint distribution of all observables, given assumptions of conditional independence among CPTs: 𝑃(𝑋1 , … , 𝑋𝑛 ) = ∏𝑠∈𝑉 𝑃(𝑋𝑠 |𝑝𝑎𝑟𝑒𝑛𝑡𝑠(𝑋𝑠 )) (1)

Which means that the conditional relationships between nodes independent given a parent node. Using latent variable modeling terminology, the exogenous variables (the source variables) are called parents and the endogenous variables (the destination of the edges) are called children. Method Sample The entire sample included N=2,110 total number of students were assessed in Grades 4. A total N=113 students were excluded from the analysis due to missing data on more than 95% of scorable items. Based on the 2009 Science Framework, each student was assigned at least one of three interactive computer tasks Mystery Plants, Here Comes the Sun or Cracking Concrete. Measures For this particular study, only students who were administered the Grade 4 Mystery Plants ICT (U.S. Department of Education, see Figure 2) were analyzed. In this ICT students use a simulated greenhouse to determine the best amount of sunlight or fertilizer two different plants need. For each simulation (sunlight or fertilizer) students are first asked to predict how much sunlight or fertilizer is required for optimal plant growth. The student then proceeds to the observation stage where he or she places six different trays of the same plant type in each of the three conditions in the virtual greenhouse, to finally draw conclusions about the simulated results. Analytical procedures Preliminary descriptive analysis of the POE item was performed in Excel 2007 to obtain counts of students who fell into patterns of POE response sets. After a descriptive analysis, a BN was drawn via edges and nodes in Netica Version 5.13 (2013) and CPTs learned via the Expectation Maximization method as proposed by Lauritzen (1995), where the CPTs are learned given a network structure and data. For the purpose of this analysis, CPTs for P, O and E nodes were defined to be uniform Dirichlet priors, although stronger priors can be implemented if the cognitive scientists has strong beliefs about prior knowledge. Preliminary results Figure 3 depicts representative skill matrices of two POE response sets. The results show that 2% of the students in our sample incorrectly predicted, observed and explained; whereas 29% of the sample incorrectly predicted, correctly observed and correctly explained. A pattern of (0,0,0) who obtained the lowest profanely levels of learning whereas a pattern of (0,1,1) then corresponds to students with the high proficiency of learning. The latter students were the once who had an incorrect assumption but performed the science laboratory, learned from the results and later explained what they learned. The BN modeling (Figure 4) shows the posterior predictive probabilities of the overall learned net. The nodes can be partitioned using conditional probabilities for P(Predict), P(Observe | Predict) and P(Explain | Predict, Observe).

Among all fourth graders taking the Mystery Plants task, the posterior probability of correctly prediction is about 0.572 the posterior predictive probability of observing correctly is 0.789, and the posterior predictive probability (PPP) of explaining correctly is 0.927. Figure 5 shows the instantiated nodes for Predict and Observe for two patterns as described in Figure 4, notably P(Explain | Predict=0, Observe=1) and P(Explain | Predict=0, Observe=0). As expected for the high learning group, those who predicted incorrectly but observed correctly received a PPP to explain of 0.948, whereas those who predicted incorrectly and observed incorrectly have a PPP of 0.848. This allows us to create expected achievement profiles for students as they progress on the task. Discussion The psychometric challenges of modeling observables in science interactive tasks lie in the fact that there may be complex conditional relationships among items. The Bayesian network is one proposed solution to this modeling challenge. This paper demonstrates that it is possible to use Bayesian networks to generate profiles of student achievement in POE scientific inquiry tasks that validate assumptions about student learning. Research is ongoing on the specification of incorporating task features beyond POE item sets that can further speak to the modeling of student proficiency.

Figure 1. Item response theory model framed as a directed acyclic graph.

Figure 2. Virtual greenhouse in the 2009 NAEP Science ICT Grade 4 Mystery Plants task.

Figure 3. POE item set response patterns of students who exhibit high learning and low learning through scientific inquiry.

Figure 4. Dynamic Bayes Net of student scientific inquiry skills through the Predict, Observe, Explain observables.

Figure 5. Instantiation of the Dynamic Bayes net conditioned on participations instantiated on patterns (0,1) and (0,0)

References Champagne, A., Bergin, K., Bybee, R., Duschl, R., & Gallagher, J. (2004). NAEP 2009 science framework development: Issues and recommendations. Paper commissioned by the National Assessment Governing Board, Washington, DC. Lauritzen, S. and Spiegelhalter, D. (1988). Local computations with probabilities on graphical structures and their application to expert systems. Journal of the Royal Statistical Society. ldots, [online] 50(2), pp.157--224. Available at: http://www.jstor.org/stable/2345762. Lauritzen, S. L. (1995). The EM algorithm for graphical association models with missing data. Computational Statistics & Data Analysis, 19(2), 191-201. Levy, R. and Mislevy, R. (2004). Specifying and Refining a Measurement Model for a SimulationBased Assessment. CSE Report 619. US Department of Education, [online] 1522(310). Available at: http://eric.ed.gov/?id=ED483385. Mislevy, R. J., Almond, R. G., Yan, D., & Steinberg, L. S. (1999, July). Bayes nets in educational assessment: Where the numbers come from. In Proceedings of the fifteenth conference on uncertainty in artificial intelligence (pp. 437-446). Morgan Kaufmann Publishers Inc. Netica. (2013). Netica-J Reference Manual—Version 5.13, Java Version of Netica API. Vancouver, Canada: Norsys Software Corporation.

U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics (2009). Science Framework for the 2009 National Assessment of Educational Progress.