Inquiries into Exploration Behavior in Complex ...

2 downloads 0 Views 4MB Size Report
Jonas C. Müller, André Kretzschmar & Samuel Greiff. University of Luxembourg. Empirical Examination: ▫. 565 students, 54% female, 14.95 years mean age ...
Exploring Exploration Inquiries into Exploration Behavior in Complex Problem Solving Assessment Jonas C. Müller, André Kretzschmar & Samuel Greiff University of Luxembourg ABSTRACT: Complex Problem Solving (CPS) is a prominent representative of transversal, domain-general skills, empirically connected to a broad range of outcomes and recently included in large-scale assessments such as PISA 2012. Advancements in the assessment of CPS are now calling for a) broader assessment vehicles allowing the whole breadth of the concept to unfold and b) additional efforts with regard to the exploitation of log-file data available. We explored the consequences of heterogeneous tasks with regard to the applicability of an established measure of strategic behavior (VOTAT) featured currently in assessment instruments. We developed a modified conception of this strategy suitable for a broader range of tasks and tested its utility on an empirical basis. Additional value is investigated along the line of theory driven educational data mining of process data. An extension of VOTAT (vary-one-thing-at-atime, also called CVS): Nested-VOTAT: n  Identifying „areas of similar functioning“ n  Sets of states: Input variations lead to comparable variations in outputs n  Systematically exploring transitions between these areas n  Scoring on a per-item basis

Items based on Finite State Automata: VOTAT not sufficient, essential states are not explored

N e s t e d - V O TAT: Exploring „areas of similar functioning“

Results n  Separable dimension for the application of nested-VOTAT in addition to the established dimensions of knowledge acquisition and knowledge application; latent factors correlated r = .71 to .81, all p < .001 n  Good model fit of three dimensional model (χ2 (167) = 230.246, p < .001, RMSEA = 0.026, CFI = 0.981, TLI = 0.978), conflating any two dimensions resulted in significantly worse model fit n  But: Poor internal consistency for nested-VOTAT indicator (Cronbach’s alpha α = .53 )

Items based on Linear Structural Equations: VOTAT sufficient Empirical Examination: n  565 students, 54% female, 14.95 years mean age (SD = 1.30) n  Eight LSE-based items (VOTAT-based strategy scoring) n  Five FSA-based items (nested-VOTAT-based strategy scoring)

VOTAT: Systematically varying all variables singularly (vary-onething-at-a-time, also called CVS)

6th International Conference on Educational Data Mining, July 6 - July 9 2013, Memphis, Tennessee, USA

Discussion and outlook n  An advance into utilizing process data in FSA-based tasks n  Reasons for poor internal consistency of strategy indicator? n  Measurement problems vs. differences in items? n  Results highlight potential of FSA-based items in CPS assessment n  Higher variation in strategy usage necessary n  Potential of using nested-VOTAT like scoring procedures as input in processes in EDM (e.g. text-replay tagging) n  Currently under development Contact: [email protected]