Column borrow to subtract the second numerator from the first. 7. Subtract numerators. 8. Reduce answers to simplest form. I Dichotomous attribute, constrained ...
Generalized Linear Mixed Proficiency Models Jonathan L. Templin Department of Psychology University of Illinois at Urbana-Champaign
Thesis Defense
Slide 1 of 36
Overview
● Overview
n
● Modeling Comparison ● Thesis Model ● Thesis Structural Model ● Thesis Measurement Model ● Thesis Objectives ● Estimation Algorithm ● Evaluation Criteria ● Proficiency Space Studies
n
● Polytomous Attributes ● Polytomous Attribute Studies ● GLMPM Application
n
● Future Research Directions ● Acknowledgments
n n
Thesis Defense
Synopsis of aims and features of thesis research: u Structural model for mixed-type (continuous and discrete) latent variables: n Generalized Linear Mixed Proficiency Model. Dichotomous attribute GLMPM simulation studies. Generalization to polytomous attributes: u Measurement model for mixed-type (continuous and multi-level discrete) latent variables: n Polytomous Attribute Reparameterized Unified Model. Polytomous attribute simulation studies. Data application with dichotomous and polytomous attributes.
Slide 2 of 36
Traditional Latent Models
● Overview
n
● Modeling Comparison ● Thesis Model ● Thesis Structural Model ● Thesis Measurement Model ● Thesis Objectives ● Estimation Algorithm ● Evaluation Criteria ● Proficiency Space Studies ● Polytomous Attributes ● Polytomous Attribute Studies
n
Structural Model: Models characteristics (primarily covariance structure) of distribution of continuous latent variables. Measurement Model: Compensatory (OR) model relating observable responses to set of continuous latent traits. Structural Model
● GLMPM Application ● Future Research Directions ● Acknowledgments
Measurement Model
Thesis Defense
Slide 3 of 36
Cognitive Diagnosis Measurement Model
● Overview
n
● Modeling Comparison ● Thesis Model ● Thesis Structural Model
n
● Thesis Measurement Model ● Thesis Objectives ● Estimation Algorithm ● Evaluation Criteria ● Proficiency Space Studies ● Polytomous Attributes ● Polytomous Attribute Studies ● GLMPM Application
n
Non-compensatory/Conjunctive (AND) model. Most cognitive diagnosis models (such as the DINA and NIDA) parameterize discrete (binary) latent traits only. The RUM models both discrete (α) and continuous (θ) latent traits. Discrete Latent Variable
“AND” Relation
● Future Research Directions ● Acknowledgments
Cognitive Diagnosis Measurement Model
Thesis Defense
Slide 4 of 36
Mixed Latent Structural Model
● Overview
n
● Modeling Comparison ● Thesis Model ● Thesis Structural Model ● Thesis Measurement Model ● Thesis Objectives ● Estimation Algorithm ● Evaluation Criteria ● Proficiency Space Studies ● Polytomous Attributes
n
Structural model for characteristics of distribution of both discrete (α) and continuous (θ) latent traits. Structural model often includes structure for mean of discrete traits (such as de la Torre and Douglas, in press; Hartz, 2002).
● Polytomous Attribute Studies ● GLMPM Application ● Future Research Directions ● Acknowledgments
Covariates
Mixed Latent Structural Model Discrete Latent Variable
Thesis Defense
Continuous Latent Variable
Mean Structures
Slide 5 of 36
Thesis Model Features
● Overview
n
● Modeling Comparison ● Thesis Model ● Thesis Structural Model ● Thesis Measurement Model ● Thesis Objectives
n
● Estimation Algorithm ● Evaluation Criteria ● Proficiency Space Studies ● Polytomous Attributes ● Polytomous Attribute Studies ● GLMPM Application ● Future Research Directions ● Acknowledgments
n
n
Thesis Defense
Mixed-type latent traits (continuous, binary-discrete, polytomous-discrete). Structural model: u Higher order covariance structure model. u Discrete latent trait mean structure model. n Mean structure models incorporate observable covariates. Measurement model: u For binary-discrete/continuous latent spaces: Reparameterized Unified Model (Hartz, 2002). u For polytomous-discrete/continuous latent spaces: Generalization of RUM. Although RUM is used as the measurement model, the GLMPM does not necessitate the use of the RUM.
Slide 6 of 36
Thesis Structural Model The Generalized Linear Mixed Proficiency Model (GLMPM) n ● Overview
n
● Modeling Comparison ● Thesis Model ● Thesis Structural Model ● Thesis Measurement Model ● Thesis Objectives ● Estimation Algorithm ● Evaluation Criteria
n n
k = 1, . . . , K latent traits (continuous - θ, discrete - α). Higher order latent trait G ∼ N (0, 1). Factor loading parameters λ (one-factor model). Fixed observable covariates Y. u
● Proficiency Space Studies ● Polytomous Attributes ● Polytomous Attribute Studies ● GLMPM Application ● Future Research Directions ● Acknowledgments
u
Conditional distribution for continuous latent traits: h(θik |yi , gi ) ∼ N (β k yi0 + λk gi , 1 − λ2k )
Conditional distribution for discrete latent traits (l = 0, 1, . . . , Lk ) is multinomial with probability of attaining level l or greater: 0 β k y√ i +λk gi −κkl P (αik ≥ l|yi , gi ) = Φ 2 1−λk
n n n
Thesis Defense
κk0 = −∞ κk(L+1) = ∞ κk0 < κk1 < . . . < κkL
Slide 7 of 36
GLMPM Illustrated
0.4
Discrete Latent Traits
0.4
Continuous Latent Traits
0.3
κk2
P(α ik = 1 |y i, g i)
P(α ik = 2 |y i, g i)
0.1
P(α ik = 0 |y i, g i)
0.0 −3
−2
−1
0 θ ik
Thesis Defense
0.2
~ |y , g ) h(α ik i i
0.2 0.0
0.1
h(θ ik |y i, g i)
0.3
κk1
1
2
3
−3
−2
−1
0
1
2
3
~ α ik
Slide 8 of 36
Reparameterized Unified Model (Hartz, 2002)
● Overview ● Modeling Comparison
P (Xij = 1|αi , θi , qj ) = Pcj (θi ) πj∗
● Thesis Model ● Thesis Structural Model
k=1
● Thesis Measurement Model ● Thesis Objectives ● Estimation Algorithm
n
● Evaluation Criteria ● Proficiency Space Studies ● Polytomous Attributes ● Polytomous Attribute Studies
n
● GLMPM Application ● Future Research Directions ● Acknowledgments
n
n
n
Thesis Defense
K Y
(1−αik )×qjk
rjk
!
Where qj is the pre-specified row vector (1 × K) of Q-matrix entries for item j. 1.701(c +θ ) j i e With “completeness” term Pcj (θi ) = 1.701(cj +θi ) 1+e
πj∗ is the maximum probability of correct response conditional on mastery of all Q-matrix attributes for item j. ∗ rjk is the “penalty” imposed for missing attribute k, meaning the maximum probability of correct response for item j is ∗ πj∗ rjk . PK Let qj· = k=1 qjk . The RUM places 2qj· equality constraints on the 2K class item response probabilities.
Slide 9 of 36
RUM Illustrated High Cog. Structure, Low Completeness 1.0 0.8 −2
0
2
4
−4
−2
0
2
4
Low Cog. Structure, High Completeness
Low Cog. Structure, Low Completeness
0.8 0.6 0.4 0.0
0.0
0.2
0.4
0.6
P(X ij =1|α i)
0.8
1.0
θi
1.0
θi
0.2
P(X ij =1|α i)
0.6 0.2 0.0
−4
−4
−2
0 θi
Thesis Defense
α = 11 α = 10 α = 01 α = 00
0.4
P(X ij =1|α i)
0.6 0.4 0.0
0.2
P(X ij =1|α i)
0.8
High Cog. Structure, High Completeness 1.0
Consider an item requiring two attributes (qj· = 2):
2
4
−4
−2
0
2
4
θi
Slide 10 of 36
Thesis Model Path Diagram
Structural Model GLMPM Covariates
Measurement Model RUM
Thesis Defense
Slide 11 of 36
Thesis Objectives n ● Overview ● Modeling Comparison ● Thesis Model ● Thesis Structural Model ● Thesis Measurement Model
n
● Thesis Objectives ● Estimation Algorithm ● Evaluation Criteria ● Proficiency Space Studies
n
● Polytomous Attributes ● Polytomous Attribute Studies ● GLMPM Application ● Future Research Directions ● Acknowledgments
n
Thesis Defense
Test the efficacy of estimation algorithm at recovery of structural (GLMPM) and measurement (RUM) model parameters across wide variety of conditions. Test robustness of both mean structure and measurement models when correlational structure is not modeled. Measurement model (RUM) conditions: u Eight attributes, 40 items. u Q-matrix complexity (low and high). u Item parameter cognitive structure (low and high). u Item completeness (low and high). Structural model (GLMPM) conditions: u β = 0; λ = 0; κ = 0 (Unstructured Proficiency Space). u β = 0; λ = 0 (Independent Attribute Effects). u β = 0 (Higher Order Latent Proficiency Space). u λ = 0 (Independent Attribute, Associated Covariate). u No constraints (Full GLMPM).
Slide 12 of 36
Estimation Algorithm
● Overview ● Modeling Comparison ● Thesis Model ● Thesis Structural Model ● Thesis Measurement Model ● Thesis Objectives ● Estimation Algorithm ● Evaluation Criteria ● Proficiency Space Studies ● Polytomous Attributes ● Polytomous Attribute Studies ● GLMPM Application ● Future Research Directions
n
MCMC estimation algorithm. u Uniform prior for all RUM item parameters (π ∗ , r ∗ , c). u Latent traits (α, θ) modeled with prior defined by GLMPM (varies by experimental condition). u Gibbs sampling step for each α parameter. u Uniform prior for all mean and covariance structural parameters (κ, λ, β).
● Acknowledgments
Thesis Defense
Slide 13 of 36
Evaluation Criteria
● Overview
n
● Modeling Comparison ● Thesis Model ● Thesis Structural Model ● Thesis Measurement Model ● Thesis Objectives ● Estimation Algorithm ● Evaluation Criteria ● Proficiency Space Studies
n
● Polytomous Attributes ● Polytomous Attribute Studies ● GLMPM Application
n
● Future Research Directions ● Acknowledgments
n
n
Thesis Defense
Systematic asymmetry in estimated parameter deviation measured by proportion of estimates being less than their true values (defined as bias measure in thesis, omitted for defense). Mean deviation is presented as a measure of bias. Estimated parameter accuracy measured by Mean Absolute Deviation. Discrete attribute accuracy on individual attribute level measured by correct classification rate, Cohen’s Kappa. Overall attribute pattern accuracy measured by average correct classification rate and whole-pattern correct classification.
Slide 14 of 36
Full GLMPM-Efficacy Study Low Complexity, High Cognitive Structure, High Completeness π*
3.0 2.5 2.0
Estimated
0.3
1.0 0.8
0.9
1.0
0.0
0.1
0.2
0.3
0.4
1.0
2.0
True
True
κ
λ
β
0.4
0.4
0.6
3.0
0.0 −0.2
0.4 0.2
2.5
0.2
Estimated
0.8
Estimated
0.6
0.4 0.2 0.0
0.0
0.4
0.5
0.6
0.7 True
G
θ
0.8
0.9
1.0
−0.2
0.0
0.2
0.4
0.6
True
2 0 −2 −4
−4
−2
0
Estimated
2
4
True
4
−0.2
1.5
True
−0.4
Estimated
0.2
Estimated
0.1 0.0 0.7
1.0
0.6
−0.4
Estimated
1.5
1.0 0.9 0.8 0.7
Estimated
0.6 0.5 0.5
−4
−2
0 True
Thesis Defense
c
0.4
r*
2
4
−4
−2
0
2
4
True
Slide 15 of 36
Full GLMPM-Efficacy Study
● Overview ● Modeling Comparison ● Thesis Model
Bias Condition
π∗
r∗
c
κ
β
λ
θ
G
● Thesis Structural Model
HHH
0.009
0.003
-0.301
-0.047
0.008
-0.013
0.050
-0.020
● Thesis Measurement Model
HHL
-0.007
0.023
0.134
-0.060
-0.113
0.012
-0.013
-0.025
HLH
0.007
0.016
-0.445
-0.088
-0.031
-0.028
0.075
-0.016
● Evaluation Criteria
HLL
-0.030
0.018
0.269
0.037
0.039
-0.008
-0.057
-0.007
● Proficiency Space Studies
LHH
0.001
0.002
-0.425
0.000
0.067
0.011
0.143
-0.014
LHL
-0.014
0.015
0.390
0.031
-0.027
0.022
-0.059
0.017
● GLMPM Application
LLH
-0.001
0.006
-0.268
0.045
0.082
-0.030
0.101
0.029
● Future Research Directions
LLL
-0.018
0.022
0.234
0.140
0.104
0.011
-0.062
0.013
● Thesis Objectives ● Estimation Algorithm
● Polytomous Attributes ● Polytomous Attribute Studies
● Acknowledgments
MAD π∗
r∗
c
κ
β
λ
θ
G
HHH
0.016
0.036
0.384
0.074
0.029
0.084
0.816
0.780
HHL
0.015
0.045
0.174
0.091
0.039
0.120
0.833
0.833
HLH
0.023
0.058
0.609
0.109
0.059
0.093
0.855
0.829
HLL
0.039
0.081
0.390
0.183
0.099
0.114
0.804
0.787
LHH
0.015
0.023
0.507
0.063
0.036
0.100
0.840
0.825
LHL
0.028
0.039
0.422
0.081
0.038
0.105
0.787
0.805
LLH
0.020
0.050
0.429
0.116
0.052
0.126
0.786
0.803
LLL
0.031
0.069
0.287
0.196
0.059
0.170
0.793
0.807
Condition
Condition Key: Complexity / Cognitive Structure / Completeness
Thesis Defense
Slide 16 of 36
Full GLMPM-Efficacy Study
● Overview ● Modeling Comparison
Classification Rate
● Thesis Model
Condition
● Thesis Structural Model ● Thesis Measurement Model
Whole
A1
A2
A3
A4
A5
A6
A7
A8
Average
Pattern
HHH
0.939
0.958
0.934
0.939
0.943
0.944
0.960
0.968
0.948
0.747
HHL
0.917
0.923
0.913
0.917
0.931
0.926
0.927
0.935
0.924
0.653
● Estimation Algorithm
HLH
0.870
0.889
0.847
0.778
0.847
0.847
0.907
0.875
0.858
0.355
● Evaluation Criteria
HLL
0.850
0.812
0.832
0.755
0.811
0.876
0.835
0.857
0.828
0.336
● Proficiency Space Studies
LHH
0.947
0.983
0.939
0.959
0.965
0.963
0.973
0.949
0.960
0.746
● Polytomous Attribute Studies
LHL
0.951
0.879
0.921
0.959
0.938
0.917
0.891
0.917
0.922
0.595
● GLMPM Application
LLH
0.878
0.896
0.845
0.880
0.896
0.823
0.899
0.892
0.876
0.397
● Future Research Directions
LLL
0.852
0.821
0.833
0.805
0.857
0.817
0.897
0.772
0.832
0.296
● Thesis Objectives
● Polytomous Attributes
● Acknowledgments
Cohen’s κ Condition
A1
A2
A3
A4
A5
A6
A7
A8
HHH
0.857
0.893
0.812
0.836
0.807
0.859
0.893
0.917
HHL
0.815
0.721
0.758
0.791
0.838
0.805
0.769
0.797
HLH
0.732
0.777
0.685
0.554
0.693
0.670
0.811
0.749
HLL
0.688
0.536
0.654
0.518
0.628
0.750
0.619
0.716
LHH
0.892
0.963
0.861
0.917
0.929
0.912
0.946
0.895
LHL
0.899
0.731
0.818
0.916
0.871
0.816
0.778
0.833
LLH
0.754
0.791
0.686
0.728
0.757
0.641
0.797
0.781
LLL
0.693
0.634
0.577
0.613
0.699
0.626
0.792
0.544
Condition Key: Complexity / Cognitive Structure / Completeness
Thesis Defense
Slide 17 of 36
Independent Attribute Effects-Robustness Study Low Complexity, High Cognitive Structure, High Completeness π*
c
0.7
0.8
0.9
1.0
0.0
0.1
0.2
True
True
κ
θ
0.3
0.4
0.0
0.5
1.0
1.5
2.0
2.5
3.0
True
0
Estimated
−0.2
−4
−0.8
−0.6
−2
−0.4
Estimated
0.0
2
0.2
4
0.4
0.6
0.0
0.0
0.6
0.5
0.1
0.7
1.0
1.5
Estimated
0.2
Estimated
0.8
Estimated
2.0
0.3
0.9
2.5
3.0
0.4
1.0
r*
−0.8
−0.6
−0.4
−0.2 True
Thesis Defense
0.0
0.2
0.4
−4
−2
0
2
4
True
Slide 18 of 36
Independent Attribute Effects-Robustness Study
● Overview ● Modeling Comparison ● Thesis Model
Bias Condition
π∗
r∗
c
κ
θ
● Thesis Structural Model
HHH
0.056
-0.018
-1.457
-0.352
0.045
● Thesis Measurement Model
HHL
0.028
-0.011
-0.433
-0.397
-0.116
HLH
0.054
-0.045
-1.895
-0.704
-0.019
● Evaluation Criteria
HLL
0.027
-0.065
-0.529
-0.795
-0.189
● Proficiency Space Studies
LHH
0.074
-0.019
-1.634
-0.380
0.014
LHL
0.018
-0.027
-0.345
-0.462
-0.109
● GLMPM Application
LLH
0.057
-0.040
-1.545
-0.698
-0.059
● Future Research Directions
LLL
0.003
-0.061
-0.257
-0.684
-0.105
● Thesis Objectives ● Estimation Algorithm
● Polytomous Attributes ● Polytomous Attribute Studies
● Acknowledgments
MAD π∗
r∗
c
κ
θ
HHH
0.056
0.037
1.457
0.352
0.815
HHL
0.051
0.045
0.447
0.397
0.804
HLH
0.058
0.077
1.895
0.704
0.826
HLL
0.052
0.110
0.532
0.795
0.826
LHH
0.074
0.039
1.634
0.380
0.804
LHL
0.037
0.048
0.357
0.462
0.806
LLH
0.064
0.070
1.545
0.698
0.784
LLL
0.036
0.093
0.311
0.684
0.784
Condition
Condition Key: Complexity / Cognitive Structure / Completeness
Thesis Defense
Slide 19 of 36
Independent Attribute Effects-Robustness Study
● Overview ● Modeling Comparison
Classification Rate
● Thesis Model
Condition
● Thesis Structural Model ● Thesis Measurement Model
Whole
A1
A2
A3
A4
A5
A6
A7
A8
Average
Pattern
HHH
0.869
0.853
0.768
0.745
0.789
0.856
0.871
0.927
0.835
0.537
HHL
0.747
0.870
0.677
0.639
0.776
0.800
0.745
0.889
0.768
0.419
● Estimation Algorithm
HLH
0.725
0.716
0.650
0.589
0.705
0.647
0.618
0.744
0.674
0.260
● Evaluation Criteria
HLL
0.601
0.651
0.548
0.668
0.746
0.644
0.646
0.632
0.642
0.208
● Proficiency Space Studies
LHH
0.787
0.867
0.903
0.853
0.821
0.781
0.861
0.864
0.842
0.486
● Polytomous Attribute Studies
LHL
0.709
0.707
0.773
0.910
0.853
0.731
0.723
0.727
0.767
0.413
● GLMPM Application
LLH
0.674
0.625
0.717
0.584
0.781
0.700
0.695
0.743
0.690
0.232
● Future Research Directions
LLL
0.608
0.675
0.689
0.666
0.675
0.744
0.736
0.597
0.673
0.220
● Thesis Objectives
● Polytomous Attributes
● Acknowledgments
Cohen’s κ Condition
A1
A2
A3
A4
A5
A6
A7
A8
HHH
0.708
0.683
0.435
0.424
0.522
0.701
0.715
0.853
HHL
0.382
0.735
0.224
0.190
0.510
0.572
0.455
0.777
HLH
0.400
0.451
0.091
0.262
0.442
0.260
0.196
0.516
HLL
0.032
0.267
0.204
0.373
0.498
0.045
0.189
0.294
LHH
0.547
0.732
0.807
0.709
0.652
0.530
0.696
0.729
LHL
0.286
0.341
0.546
0.820
0.702
0.433
0.411
0.328
LLH
0.250
0.320
0.465
0.128
0.579
0.427
0.333
0.494
LLL
0.038
0.380
0.392
0.306
0.114
0.486
0.398
0.052
Condition Key: Complexity / Cognitive Structure / Completeness
Thesis Defense
Slide 20 of 36
Discussion
● Overview
n
● Modeling Comparison ● Thesis Model ● Thesis Structural Model ● Thesis Measurement Model ● Thesis Objectives ● Estimation Algorithm ● Evaluation Criteria ● Proficiency Space Studies ● Polytomous Attributes ● Polytomous Attribute Studies ● GLMPM Application ● Future Research Directions ● Acknowledgments
n
Thesis Defense
Item parameter results indicated good recovery in efficacy study, poor recovery in robustness study. u Estimation of π ∗ and r ∗ parameters was generally accurate in efficacy studies. u c parameter was estimated with most difficulty, particularly in cases with high completeness. u In robustness studies, c parameter was nearly always under-estimated. Estimation of the measurement model latent traits varied by trait type. u Classification rates were considerably above chance in efficacy studies. u Classification rate plummeted in robustness studies. u Similar to Hartz (2002), estimation of θ parameter was difficult in almost all cases (although is better in robustness studies). Slide 21 of 36
Discussion (Continued)
● Overview
n
● Modeling Comparison ● Thesis Model ● Thesis Structural Model ● Thesis Measurement Model ● Thesis Objectives
n
● Estimation Algorithm ● Evaluation Criteria ● Proficiency Space Studies ● Polytomous Attributes
n
● Polytomous Attribute Studies ● GLMPM Application ● Future Research Directions ● Acknowledgments
n
n
Thesis Defense
In efficacy studies, structural model parameters were accurately estimated with a limited amount of bias. GLMPM is accurately estimated across a wide variety of RUM conditions when proficiency space is modeled correctly. Mean structure and measurement model parameter estimates are not robust when correlational structure is not modeled. Severe lack of robustness when correlation is not modeled, indicating that correlation should always be parameterized. Effects of not modeling correlation were not limited to structural model/proficiency space parameters, but were shown to cause extreme bias in c parameter estimates.
Slide 22 of 36
Generalization to Polytomous Attributes
● Overview ● Modeling Comparison
P (Xij = 1|αi , θi , qj ) = Pcj (θi ) πj∗
● Thesis Model ● Thesis Structural Model ● Thesis Measurement Model ● Thesis Objectives ● Estimation Algorithm
● Polytomous Attributes
j +θi ) e 1+e1.701(cj +θi )
n
For dichotomous attributes (Hartz, 2002): ujk (αik , qjk ) = (1 − αik ) × qjk For polytomous attributes: ujk (αik = l, qjk ) = fjkl × qjk , such that: fjk0 = 1 > fjk1 > . . . > fjkL = 0 Constrained polytomous attribute RUM (single additional parameter per attribute level l): f1kl = f2kl = . . . = fJkl GLMPM is generalized by incorporating ordered κkl parameters for each attribute.
● GLMPM Application ● Future Research Directions
n
n
n
Thesis Defense
k=11.701(c
(αik ,qjk )
With “completeness” term: Pcj (θi ) =
● Polytomous Attribute Studies
● Acknowledgments
∗u
rjk jk
!
n
● Evaluation Criteria ● Proficiency Space Studies
K Y
Slide 23 of 36
Polytomous Attribute RUM Illustrated 1.0
π*j = 0.90 | r*j1 = 0.30 | r*j2 = 0.10 | fj11 = 0.50 | fj21 = 0.75 | cj = 2.50
0.6 0.4 0.0
0.2
P(X ij =1|α i)
0.8
α = 22 α = 21 α = 20 α = 12 α = 11 α = 10 α = 02 α = 01 α = 00
−4
−2
0
2
4
θi
Thesis Defense
Slide 24 of 36
Polytomous Attribute Studies
● Overview
n
● Modeling Comparison ● Thesis Model ● Thesis Structural Model ● Thesis Measurement Model ● Thesis Objectives ● Estimation Algorithm
n
● Evaluation Criteria ● Proficiency Space Studies ● Polytomous Attributes ● Polytomous Attribute Studies ● GLMPM Application
n
● Future Research Directions ● Acknowledgments
n
Thesis Defense
Test the efficacy of estimation algorithm at recovery of structural (GLMPM) and measurement (Polytomous Attribute RUM) model parameters across varying conditions. Two attributes trichotomous (0, 1, 2), six attributes dichotomous (0, 1). Measurement model (RUM) conditions: u Q-matrix complexity (low and high). u Item parameter cognitive structure (low and high). u Item completeness (low and high). Structural model (GLMPM) condition: u Higher Order Latent Proficiency Space - β = 0.
Slide 25 of 36
Unconstrained Polytomous Attribute Studies Low Complexity, High Cognitive Structure, High Completeness π*
0.8
0.9
1.0
0.6 0.4 0.0
0.2
Estimated 0.0
0.1
0.2
0.3
0.4
0.0
0.1
0.2
0.3
True
True
c
κ
λ
2.0
2.5
3.0
0.5
0.6
0.8
Estimated
0.4
0.6
1.0 0.0 −1.0
−0.5
Estimated
0.5
2.5 2.0 1.5
1.5
0.4
1.0
True
1.0
−1.0
−0.5
0.0 True
G
θ
0.5
1.0
0.4
0.5
0.6
0.7
0.8
0.9
1.0
True
2 −2 −4
−4
−2
0
Estimated
2
4
True
0
Estimated
0.3 0.1 0.0
0.7
4
1.0
Estimated
0.2
Estimated
0.9 0.8 0.7
Estimated
0.6 0.5
0.6
3.0
0.5
−4
−2
0 True
Thesis Defense
f
0.4
1.0
r*
2
4
−4
−2
0
2
4
True
Slide 26 of 36
Unconstrained Polytomous Attribute Studies
● Overview ● Modeling Comparison ● Thesis Model
Bias Condition
π∗
r∗
c
f
κ
λ
θ
G
● Thesis Structural Model
HHH
0.004
0.007
-0.271
0.005
-0.033
-0.003
-0.037
-0.013
● Thesis Measurement Model
HHL
-0.003
0.016
0.217
0.038
0.046
-0.012
-0.085
0.020
HLH
0.009
0.010
-0.489
0.106
-0.091
-0.034
-0.300
0.032
● Evaluation Criteria
HLL
0.000
-0.018
0.306
0.150
-0.101
0.021
0.158
-0.014
● Proficiency Space Studies
LHH
0.006
0.007
-0.191
0.042
-0.007
-0.017
-0.170
0.008
LHL
-0.016
0.010
0.082
-0.026
0.023
-0.008
-0.052
0.026
● GLMPM Application
LLH
-0.005
-0.012
-0.041
0.153
-0.186
-0.035
-0.334
-0.046
● Future Research Directions
LLL
-0.019
0.009
0.151
0.095
-0.035
0.034
0.292
-0.004
● Thesis Objectives ● Estimation Algorithm
● Polytomous Attributes ● Polytomous Attribute Studies
● Acknowledgments
MAD π∗
r∗
c
f
κ
λ
θ
G
HHH
0.022
0.030
0.460
0.048
0.044
0.026
0.645
0.860
HHL
0.022
0.041
0.276
0.096
0.060
0.057
0.494
0.390
HLH
0.023
0.059
0.616
0.154
0.173
0.066
0.633
0.438
HLL
0.024
0.074
0.373
0.189
0.182
0.080
0.507
0.436
LHH
0.021
0.026
0.457
0.094
0.050
0.060
0.588
0.430
LHL
0.025
0.031
0.138
0.096
0.065
0.031
0.386
0.385
LLH
0.020
0.038
0.323
0.206
0.194
0.046
0.643
0.424
LLL
0.036
0.079
0.239
0.157
0.162
0.074
0.453
0.476
Condition
Condition Key: Complexity / Cognitive Structure / Completeness
Thesis Defense
Slide 27 of 36
Unconstrained Polytomous Attribute Studies
● Overview ● Modeling Comparison
Classification Rate
● Thesis Model
Condition
● Thesis Structural Model ● Thesis Measurement Model
Whole
A1
A2
A3
A4
A5
A6
A7
A8
Average
Pattern
HHH
0.767
0.803
0.887
0.864
0.928
0.915
0.934
0.956
0.882
0.439
HHL
0.704
0.794
0.868
0.858
0.880
0.888
0.891
0.925
0.851
0.374
● Estimation Algorithm
HLH
0.572
0.680
0.864
0.818
0.802
0.855
0.864
0.904
0.795
0.202
● Evaluation Criteria
HLL
0.622
0.541
0.813
0.818
0.882
0.850
0.836
0.848
0.776
0.194
● Proficiency Space Studies
LHH
0.770
0.808
0.950
0.945
0.933
0.926
0.960
0.906
0.900
0.444
● Polytomous Attribute Studies
LHL
0.666
0.720
0.889
0.901
0.919
0.878
0.926
0.899
0.850
0.331
● GLMPM Application
LLH
0.618
0.564
0.891
0.878
0.938
0.930
0.870
0.892
0.823
0.230
● Future Research Directions
LLL
0.569
0.486
0.780
0.819
0.836
0.840
0.820
0.820
0.746
0.127
● Thesis Objectives
● Polytomous Attributes
● Acknowledgments
Cohen’s κ Condition
A1
A2
A3
A4
A5
A6
A7
A8
HHH
0.631
0.690
0.763
0.722
0.857
0.826
0.859
0.909
HHL
0.549
0.683
0.737
0.718
0.754
0.766
0.768
0.847
HLH
0.358
0.502
0.702
0.621
0.602
0.705
0.724
0.800
HLL
0.422
0.346
0.602
0.620
0.749
0.693
0.669
0.673
LHH
0.648
0.659
0.900
0.890
0.866
0.846
0.917
0.808
LHL
0.473
0.556
0.773
0.792
0.838
0.752
0.851
0.794
LLH
0.412
0.372
0.782
0.742
0.871
0.851
0.703
0.785
LLL
0.347
0.268
0.560
0.639
0.648
0.670
0.604
0.593
Condition Key: Complexity / Cognitive Structure / Completeness
Thesis Defense
Slide 28 of 36
Discussion
● Overview
n
● Modeling Comparison ● Thesis Model ● Thesis Structural Model ● Thesis Measurement Model ● Thesis Objectives ● Estimation Algorithm ● Evaluation Criteria ● Proficiency Space Studies ● Polytomous Attributes ● Polytomous Attribute Studies ● GLMPM Application ● Future Research Directions ● Acknowledgments
n
n
Thesis Defense
Item parameter results indicated generally good recovery of true parameters. u Estimation of π ∗ and r ∗ parameters was accurate. u c parameter was estimated with most difficulty, particularly in cases with high completeness. u f parameter recovery was generally accurate for both constrained and unconstrained models. Estimation of the measurement model latent traits varied by trait type. u Classification rates were typically well above chance, although reduced for polytomous attributes. u Estimation of θ parameter was difficult in almost all conditions. Structural model parameters were accurately estimated with a limited amount of bias.
Slide 29 of 36
Illustrative Application
● Overview
n
● Modeling Comparison ● Thesis Model ● Thesis Structural Model ● Thesis Measurement Model ● Thesis Objectives ● Estimation Algorithm
Fraction Subtraction data set. u 20 item Math test given to 2,144 middle school students. u Q-matrix has eight attributes (average 2.75 attributes per item):
● Evaluation Criteria
1. Convert a whole number to a fraction.
● Proficiency Space Studies
2. Separate a whole number from fraction.
● Polytomous Attributes
3. Simplify before subtracting.
● Polytomous Attribute Studies
4. Find a common denominator.
● GLMPM Application ● Future Research Directions
5. Borrow from whole number part.
● Acknowledgments
6. Column borrow to subtract the second numerator from the first. 7. Subtract numerators. 8. Reduce answers to simplest form.
n
n
Thesis Defense
Dichotomous attribute, constrained polytomous attribute, and unconstrained polytomous attribute models estimated. u In polytomous attribute analysis, all attributes were treated as trichotomous (0, 1, 2). Higher order proficiency space structural model.
Slide 30 of 36
Fraction Subtraction Analysis
● Overview ● Modeling Comparison ● Thesis Model ● Thesis Structural Model ● Thesis Measurement Model ● Thesis Objectives ● Estimation Algorithm ● Evaluation Criteria ● Proficiency Space Studies ● Polytomous Attributes ● Polytomous Attribute Studies ● GLMPM Application ● Future Research Directions ● Acknowledgments
Thesis Defense
n
Measurement model parameter estimates: u π ∗ and r ∗ parameter estimates were generally indicative of quality items and were consistent across models. u c parameter estimates were generally high (with few exceptions), and became higher as additional attribute levels were added. u In the constrained polytomous model, four polytomous item parameters (f ) approached zero, indicating a questionable application of polytomous attributes. u A similar phenomenon was found with many of the unconstrained polytomous model parameters approaching zero.
Slide 31 of 36
Fraction Subtraction Analysis
● Overview ● Modeling Comparison
Structural model parameter estimates: Dichotomous
● Thesis Model ● Thesis Structural Model ● Thesis Measurement Model ● Thesis Objectives
Attribute
● Estimation Algorithm
Constrained
Unconstrained
Polytomous
Polytomous
κk1
λ
κk1
κk2
λ
κk1
κk2
λ
1
-0.053
0.957
-0.292
-0.023
0.963
-0.057
0.705
0.966
2
-1.175
0.699
-1.153
0.163
0.753
-1.070
0.576
0.791
● Polytomous Attributes
3
-0.558
0.608
-3.279
-2.315
0.354
-3.446
-2.107
-0.424
● Polytomous Attribute Studies
4
-0.221
0.912
-0.266
-0.126
0.917
-0.298
-0.188
0.897
5
0.022
0.892
-0.140
0.085
0.892
-0.009
0.050
0.900
6
-0.506
0.989
-0.680
2.982
0.986
-2.711
-1.537
0.296
7
-1.009
0.969
-1.026
-0.392
0.962
-1.021
-0.386
0.994
8
0.104
0.916
-2.534
-0.030
0.839
-0.271
0.347
0.874
θ
-
0.819
-
-
0.705
-
-
0.714
● Evaluation Criteria ● Proficiency Space Studies
● GLMPM Application ● Future Research Directions ● Acknowledgments
Thesis Defense
Slide 32 of 36
Fraction Subtraction Analysis
● Overview ● Modeling Comparison
Model fit indices:
● Thesis Model
Analysis
● Thesis Structural Model ● Thesis Measurement Model ● Thesis Objectives ● Estimation Algorithm
Parameters
AIC
BIC
log(L)
Dichotomous
113
117,766.0
117,555.3
-58,770.0
Constrained Polytomous
129
112,382.2
112,139.5
-56,062.1
Unconstrained Polytomous
185
115,556.6
115,201.9
-57,593.3
● Evaluation Criteria ● Proficiency Space Studies ● Polytomous Attributes
n
● Polytomous Attribute Studies ● GLMPM Application ● Future Research Directions ● Acknowledgments
Thesis Defense
n
Fit indices calculated by Monte Carlo approximation. Constrained polytomous model is selected by both AIC and BIC.
Slide 33 of 36
Fraction Subtraction Discussion
● Overview
n
● Modeling Comparison ● Thesis Model ● Thesis Structural Model ● Thesis Measurement Model ● Thesis Objectives ● Estimation Algorithm ● Evaluation Criteria ● Proficiency Space Studies ● Polytomous Attributes ● Polytomous Attribute Studies ● GLMPM Application ● Future Research Directions ● Acknowledgments
n
Thesis Defense
Although AIC and BIC suggest otherwise, all signs point to the dichotomous attribute model as providing most intuitive explanation of parameter estimates. u Minimal changes in π ∗ , r ∗ , and c parameter estimates. u Polytomous attribute structural model κ parameters indicating some 99% of distribution concentrated on single attribute level. u Negative λ parameter in unconstrained polytomous attribute structural model. Dichotomous models have shown to be reasonable in other applications using the DINA model (de la Torre and Douglas, in press). u If DINA model fits adequately, even dichotomous RUM may be superfluous.
Slide 34 of 36
An Eye to the Future
● Overview
n
● Modeling Comparison ● Thesis Model ● Thesis Structural Model ● Thesis Measurement Model ● Thesis Objectives ● Estimation Algorithm
n
● Evaluation Criteria ● Proficiency Space Studies ● Polytomous Attributes ● Polytomous Attribute Studies ● GLMPM Application ● Future Research Directions ● Acknowledgments
n n
Thesis Defense
Evaluation of appropriateness of cognitive diagnosis models in psychological testing (such as personality assessment or clinical diagnosis). Generalization of structural model for latent trait associations. u General model for attribute (tetrachoric) correlations. u Appropriateness of structural equation model in applications. Better indices of model fit and comparison. Modifications of polytomous attribute model. u Omission of Pc (θ) term. u Parameterization of Pc (θ) as discrete polytomous attribute with Q-matrix entries on every item. u Application with other cognitive diagnosis models (such as NIDA). Slide 35 of 36
Acknowledgments
● Overview
n
● Modeling Comparison ● Thesis Model ● Thesis Structural Model ● Thesis Measurement Model ● Thesis Objectives ● Estimation Algorithm ● Evaluation Criteria
n n n
● Proficiency Space Studies ● Polytomous Attributes ● Polytomous Attribute Studies ● GLMPM Application ● Future Research Directions ● Acknowledgments
n n n n
Thesis Defense
Jeff Douglas Bill Stout Larry Hubert David Budescu Carolyn Anderson Louis Roussos Bob Henson Sara Templin
Slide 36 of 36