Generalized Linear Mixed Proficiency Models - Jonathan Templin's

Generalized Linear Mixed Proficiency Models Jonathan L. Templin Department of Psychology University of Illinois at Urbana-Champaign

Thesis Defense

Slide 1 of 36

Overview

● Overview

n

● Modeling Comparison ● Thesis Model ● Thesis Structural Model ● Thesis Measurement Model ● Thesis Objectives ● Estimation Algorithm ● Evaluation Criteria ● Proficiency Space Studies

n

● Polytomous Attributes ● Polytomous Attribute Studies ● GLMPM Application

n

● Future Research Directions ● Acknowledgments

n n

Thesis Defense

Synopsis of aims and features of thesis research: u Structural model for mixed-type (continuous and discrete) latent variables: n Generalized Linear Mixed Proficiency Model. Dichotomous attribute GLMPM simulation studies. Generalization to polytomous attributes: u Measurement model for mixed-type (continuous and multi-level discrete) latent variables: n Polytomous Attribute Reparameterized Unified Model. Polytomous attribute simulation studies. Data application with dichotomous and polytomous attributes.

Slide 2 of 36

Traditional Latent Models

● Overview

n

● Modeling Comparison ● Thesis Model ● Thesis Structural Model ● Thesis Measurement Model ● Thesis Objectives ● Estimation Algorithm ● Evaluation Criteria ● Proficiency Space Studies ● Polytomous Attributes ● Polytomous Attribute Studies

n

Structural Model: Models characteristics (primarily covariance structure) of distribution of continuous latent variables. Measurement Model: Compensatory (OR) model relating observable responses to set of continuous latent traits. Structural Model

● GLMPM Application ● Future Research Directions ● Acknowledgments

Measurement Model

Thesis Defense

Slide 3 of 36

Cognitive Diagnosis Measurement Model

● Overview

n

● Modeling Comparison ● Thesis Model ● Thesis Structural Model

n

● Thesis Measurement Model ● Thesis Objectives ● Estimation Algorithm ● Evaluation Criteria ● Proficiency Space Studies ● Polytomous Attributes ● Polytomous Attribute Studies ● GLMPM Application

n

Non-compensatory/Conjunctive (AND) model. Most cognitive diagnosis models (such as the DINA and NIDA) parameterize discrete (binary) latent traits only. The RUM models both discrete (α) and continuous (θ) latent traits. Discrete Latent Variable

“AND” Relation


Cognitive Diagnosis Measurement Model

Thesis Defense

Slide 4 of 36

Mixed Latent Structural Model

● Overview

n

● Modeling Comparison ● Thesis Model ● Thesis Structural Model ● Thesis Measurement Model ● Thesis Objectives ● Estimation Algorithm ● Evaluation Criteria ● Proficiency Space Studies ● Polytomous Attributes

n

Structural model for characteristics of distribution of both discrete (α) and continuous (θ) latent traits. Structural model often includes structure for mean of discrete traits (such as de la Torre and Douglas, in press; Hartz, 2002).

● Polytomous Attribute Studies ● GLMPM Application ● Future Research Directions ● Acknowledgments

Covariates

Mixed Latent Structural Model Discrete Latent Variable

Thesis Defense

Continuous Latent Variable

Mean Structures

Slide 5 of 36

Thesis Model Features

● Overview

n

● Modeling Comparison ● Thesis Model ● Thesis Structural Model ● Thesis Measurement Model ● Thesis Objectives

n

● Estimation Algorithm ● Evaluation Criteria ● Proficiency Space Studies ● Polytomous Attributes ● Polytomous Attribute Studies ● GLMPM Application ● Future Research Directions ● Acknowledgments

n

n

Thesis Defense

Mixed-type latent traits (continuous, binary-discrete, polytomous-discrete). Structural model: u Higher order covariance structure model. u Discrete latent trait mean structure model. n Mean structure models incorporate observable covariates. Measurement model: u For binary-discrete/continuous latent spaces: Reparameterized Unified Model (Hartz, 2002). u For polytomous-discrete/continuous latent spaces: Generalization of RUM. Although RUM is used as the measurement model, the GLMPM does not necessitate the use of the RUM.

Slide 6 of 36

Thesis Structural Model The Generalized Linear Mixed Proficiency Model (GLMPM) n ● Overview

n

● Modeling Comparison ● Thesis Model ● Thesis Structural Model ● Thesis Measurement Model ● Thesis Objectives ● Estimation Algorithm ● Evaluation Criteria

n n

k = 1, . . . , K latent traits (continuous - θ, discrete - α). Higher order latent trait G ∼ N (0, 1). Factor loading parameters λ (one-factor model). Fixed observable covariates Y. u

● Proficiency Space Studies ● Polytomous Attributes ● Polytomous Attribute Studies ● GLMPM Application ● Future Research Directions ● Acknowledgments

u

Conditional distribution for continuous latent traits: h(θik |yi , gi ) ∼ N (β k yi0 + λk gi , 1 − λ2k )

Conditional distribution for discrete latent traits (l = 0, 1, . . . , Lk ) is multinomial with probability of attaining level l or greater: 0 β k y√ i +λk gi −κkl P (αik ≥ l|yi , gi ) = Φ 2 1−λk

n n n

Thesis Defense

κk0 = −∞ κk(L+1) = ∞ κk0 < κk1 < . . . < κkL

Slide 7 of 36

GLMPM Illustrated

0.4

Discrete Latent Traits

0.4

Continuous Latent Traits

0.3

κk2

P(α ik = 1 |y i, g i)

P(α ik = 2 |y i, g i)

0.1

P(α ik = 0 |y i, g i)

0.0 −3

−2

−1

0 θ ik

Thesis Defense

0.2

~ |y , g ) h(α ik i i

0.2 0.0

0.1

h(θ ik |y i, g i)

0.3

κk1

1

2

3

−3

−2

−1

0

1

2

3

~ α ik

Slide 8 of 36

Reparameterized Unified Model (Hartz, 2002)

● Overview ● Modeling Comparison

P (Xij = 1|αi , θi , qj ) = Pcj (θi ) πj∗

● Thesis Model ● Thesis Structural Model

k=1

● Thesis Measurement Model ● Thesis Objectives ● Estimation Algorithm

n

● Evaluation Criteria ● Proficiency Space Studies ● Polytomous Attributes ● Polytomous Attribute Studies

n


n

n

n

Thesis Defense

K Y

(1−αik )×qjk

rjk

!

Where qj is the pre-specified row vector (1 × K) of Q-matrix entries for item j. 1.701(c +θ ) j i e With “completeness” term Pcj (θi ) = 1.701(cj +θi ) 1+e

πj∗ is the maximum probability of correct response conditional on mastery of all Q-matrix attributes for item j. ∗ rjk is the “penalty” imposed for missing attribute k, meaning the maximum probability of correct response for item j is ∗ πj∗ rjk . PK Let qj· = k=1 qjk . The RUM places 2qj· equality constraints on the 2K class item response probabilities.

Slide 9 of 36

RUM Illustrated High Cog. Structure, Low Completeness 1.0 0.8 −2

0

2

4

−4

−2

0

2

4

Low Cog. Structure, High Completeness

Low Cog. Structure, Low Completeness

0.8 0.6 0.4 0.0

0.0

0.2

0.4

0.6

P(X ij =1|α i)

0.8

1.0

θi

1.0

θi

0.2

P(X ij =1|α i)

0.6 0.2 0.0

−4

−4

−2

0 θi

Thesis Defense

α = 11 α = 10 α = 01 α = 00

0.4

P(X ij =1|α i)

0.6 0.4 0.0

0.2

P(X ij =1|α i)

0.8

High Cog. Structure, High Completeness 1.0

Consider an item requiring two attributes (qj· = 2):

2

4

−4

−2

0

2

4

θi

Slide 10 of 36

Thesis Model Path Diagram

Structural Model GLMPM Covariates

Measurement Model RUM

Thesis Defense

Slide 11 of 36

Thesis Objectives n ● Overview ● Modeling Comparison ● Thesis Model ● Thesis Structural Model ● Thesis Measurement Model

n

● Thesis Objectives ● Estimation Algorithm ● Evaluation Criteria ● Proficiency Space Studies

n

● Polytomous Attributes ● Polytomous Attribute Studies ● GLMPM Application ● Future Research Directions ● Acknowledgments

n

Thesis Defense

Test the efficacy of estimation algorithm at recovery of structural (GLMPM) and measurement (RUM) model parameters across wide variety of conditions. Test robustness of both mean structure and measurement models when correlational structure is not modeled. Measurement model (RUM) conditions: u Eight attributes, 40 items. u Q-matrix complexity (low and high). u Item parameter cognitive structure (low and high). u Item completeness (low and high). Structural model (GLMPM) conditions: u β = 0; λ = 0; κ = 0 (Unstructured Proficiency Space). u β = 0; λ = 0 (Independent Attribute Effects). u β = 0 (Higher Order Latent Proficiency Space). u λ = 0 (Independent Attribute, Associated Covariate). u No constraints (Full GLMPM).

Slide 12 of 36

Estimation Algorithm

● Overview ● Modeling Comparison ● Thesis Model ● Thesis Structural Model ● Thesis Measurement Model ● Thesis Objectives ● Estimation Algorithm ● Evaluation Criteria ● Proficiency Space Studies ● Polytomous Attributes ● Polytomous Attribute Studies ● GLMPM Application ● Future Research Directions

n

MCMC estimation algorithm. u Uniform prior for all RUM item parameters (π ∗ , r ∗ , c). u Latent traits (α, θ) modeled with prior defined by GLMPM (varies by experimental condition). u Gibbs sampling step for each α parameter. u Uniform prior for all mean and covariance structural parameters (κ, λ, β).

● Acknowledgments

Thesis Defense

Slide 13 of 36

Evaluation Criteria

● Overview

n

● Modeling Comparison ● Thesis Model ● Thesis Structural Model ● Thesis Measurement Model ● Thesis Objectives ● Estimation Algorithm ● Evaluation Criteria ● Proficiency Space Studies

n

● Polytomous Attributes ● Polytomous Attribute Studies ● GLMPM Application

n


n

n

Thesis Defense

Systematic asymmetry in estimated parameter deviation measured by proportion of estimates being less than their true values (defined as bias measure in thesis, omitted for defense). Mean deviation is presented as a measure of bias. Estimated parameter accuracy measured by Mean Absolute Deviation. Discrete attribute accuracy on individual attribute level measured by correct classification rate, Cohen’s Kappa. Overall attribute pattern accuracy measured by average correct classification rate and whole-pattern correct classification.

Slide 14 of 36

Full GLMPM-Efficacy Study Low Complexity, High Cognitive Structure, High Completeness π*

3.0 2.5 2.0

Estimated

0.3

1.0 0.8

0.9

1.0

0.0

0.1

0.2

0.3

0.4

1.0

2.0

True

True

κ

λ

β

0.4

0.4

0.6

3.0

0.0 −0.2

0.4 0.2

2.5

0.2

Estimated

0.8

Estimated

0.6

0.4 0.2 0.0

0.0

0.4

0.5

0.6

0.7 True

G

θ

0.8

0.9

1.0

−0.2

0.0

0.2

0.4

0.6

True

2 0 −2 −4

−4

−2

0

Estimated

2

4

True

4

−0.2

1.5

True

−0.4

Estimated

0.2

Estimated

0.1 0.0 0.7

1.0

0.6

−0.4

Estimated

1.5

1.0 0.9 0.8 0.7

Estimated

0.6 0.5 0.5

−4

−2

0 True

Thesis Defense

c

0.4

r*

2

4

−4

−2

0

2

4

True

Slide 15 of 36

Full GLMPM-Efficacy Study

● Overview ● Modeling Comparison ● Thesis Model

Bias Condition

π∗

r∗

c

κ

β

λ

θ

G

● Thesis Structural Model

HHH

0.009

0.003

-0.301

-0.047

0.008

-0.013

0.050

-0.020

● Thesis Measurement Model

HHL

-0.007

0.023

0.134

-0.060

-0.113

0.012

-0.013

-0.025

HLH

0.007

0.016

-0.445

-0.088

-0.031

-0.028

0.075

-0.016

● Evaluation Criteria

HLL

-0.030

0.018

0.269

0.037

0.039

-0.008

-0.057

-0.007

● Proficiency Space Studies

LHH

0.001

0.002

-0.425

0.000

0.067

0.011

0.143

-0.014

LHL

-0.014

0.015

0.390

0.031

-0.027

0.022

-0.059

0.017

● GLMPM Application

LLH

-0.001

0.006

-0.268

0.045

0.082

-0.030

0.101

0.029

● Future Research Directions

LLL

-0.018

0.022

0.234

0.140

0.104

0.011

-0.062

0.013

● Thesis Objectives ● Estimation Algorithm

● Polytomous Attributes ● Polytomous Attribute Studies

● Acknowledgments

MAD π∗

r∗

c

κ

β

λ

θ

G

HHH

0.016

0.036

0.384

0.074

0.029

0.084

0.816

0.780

HHL

0.015

0.045

0.174

0.091

0.039

0.120

0.833

0.833

HLH

0.023

0.058

0.609

0.109

0.059

0.093

0.855

0.829

HLL

0.039

0.081

0.390

0.183

0.099

0.114

0.804

0.787

LHH

0.015

0.023

0.507

0.063

0.036

0.100

0.840

0.825

LHL

0.028

0.039

0.422

0.081

0.038

0.105

0.787

0.805

LLH

0.020

0.050

0.429

0.116

0.052

0.126

0.786

0.803

LLL

0.031

0.069

0.287

0.196

0.059

0.170

0.793

0.807

Condition

Condition Key: Complexity / Cognitive Structure / Completeness

Thesis Defense

Slide 16 of 36

Full GLMPM-Efficacy Study


Classification Rate

● Thesis Model

Condition

● Thesis Structural Model ● Thesis Measurement Model

Whole

A1

A2

A3

A4

A5

A6

A7

A8

Average

Pattern

HHH

0.939

0.958

0.934

0.939

0.943

0.944

0.960

0.968

0.948

0.747

HHL

0.917

0.923

0.913

0.917

0.931

0.926

0.927

0.935

0.924

0.653

● Estimation Algorithm

HLH

0.870

0.889

0.847

0.778

0.847

0.847

0.907

0.875

0.858

0.355


HLL

0.850

0.812

0.832

0.755

0.811

0.876

0.835

0.857

0.828

0.336


LHH

0.947

0.983

0.939

0.959

0.965

0.963

0.973

0.949

0.960

0.746

● Polytomous Attribute Studies

LHL

0.951

0.879

0.921

0.959

0.938

0.917

0.891

0.917

0.922

0.595


LLH

0.878

0.896

0.845

0.880

0.896

0.823

0.899

0.892

0.876

0.397


LLL

0.852

0.821

0.833

0.805

0.857

0.817

0.897

0.772

0.832

0.296

● Thesis Objectives

● Polytomous Attributes

● Acknowledgments

Cohen’s κ Condition

A1

A2

A3

A4

A5

A6

A7

A8

HHH

0.857

0.893

0.812

0.836

0.807

0.859

0.893

0.917

HHL

0.815

0.721

0.758

0.791

0.838

0.805

0.769

0.797

HLH

0.732

0.777

0.685

0.554

0.693

0.670

0.811

0.749

HLL

0.688

0.536

0.654

0.518

0.628

0.750

0.619

0.716

LHH

0.892

0.963

0.861

0.917

0.929

0.912

0.946

0.895

LHL

0.899

0.731

0.818

0.916

0.871

0.816

0.778

0.833

LLH

0.754

0.791

0.686

0.728

0.757

0.641

0.797

0.781

LLL

0.693

0.634

0.577

0.613

0.699

0.626

0.792

0.544


Thesis Defense

Slide 17 of 36

Independent Attribute Effects-Robustness Study Low Complexity, High Cognitive Structure, High Completeness π*

c

0.7

0.8

0.9

1.0

0.0

0.1

0.2

True

True

κ

θ

0.3

0.4

0.0

0.5

1.0

1.5

2.0

2.5

3.0

True

0

Estimated

−0.2

−4

−0.8

−0.6

−2

−0.4

Estimated

0.0

2

0.2

4

0.4

0.6

0.0

0.0

0.6

0.5

0.1

0.7

1.0

1.5

Estimated

0.2

Estimated

0.8

Estimated

2.0

0.3

0.9

2.5

3.0

0.4

1.0

r*

−0.8

−0.6

−0.4

−0.2 True

Thesis Defense

0.0

0.2

0.4

−4

−2

0

2

4

True

Slide 18 of 36

Independent Attribute Effects-Robustness Study


Bias Condition

π∗

r∗

c

κ

θ


HHH

0.056

-0.018

-1.457

-0.352

0.045


HHL

0.028

-0.011

-0.433

-0.397

-0.116

HLH

0.054

-0.045

-1.895

-0.704

-0.019


HLL

0.027

-0.065

-0.529

-0.795

-0.189


LHH

0.074

-0.019

-1.634

-0.380

0.014

LHL

0.018

-0.027

-0.345

-0.462

-0.109


LLH

0.057

-0.040

-1.545

-0.698

-0.059


LLL

0.003

-0.061

-0.257

-0.684

-0.105



● Acknowledgments

MAD π∗

r∗

c

κ

θ

HHH

0.056

0.037

1.457

0.352

0.815

HHL

0.051

0.045

0.447

0.397

0.804

HLH

0.058

0.077

1.895

0.704

0.826

HLL

0.052

0.110

0.532

0.795

0.826

LHH

0.074

0.039

1.634

0.380

0.804

LHL

0.037

0.048

0.357

0.462

0.806

LLH

0.064

0.070

1.545

0.698

0.784

LLL

0.036

0.093

0.311

0.684

0.784

Condition


Thesis Defense

Slide 19 of 36

Independent Attribute Effects-Robustness Study


Classification Rate

● Thesis Model

Condition


Whole

A1

A2

A3

A4

A5

A6

A7

A8

Average

Pattern

HHH

0.869

0.853

0.768

0.745

0.789

0.856

0.871

0.927

0.835

0.537

HHL

0.747

0.870

0.677

0.639

0.776

0.800

0.745

0.889

0.768

0.419


HLH

0.725

0.716

0.650

0.589

0.705

0.647

0.618

0.744

0.674

0.260


HLL

0.601

0.651

0.548

0.668

0.746

0.644

0.646

0.632

0.642

0.208


LHH

0.787

0.867

0.903

0.853

0.821

0.781

0.861

0.864

0.842

0.486


LHL

0.709

0.707

0.773

0.910

0.853

0.731

0.723

0.727

0.767

0.413


LLH

0.674

0.625

0.717

0.584

0.781

0.700

0.695

0.743

0.690

0.232


LLL

0.608

0.675

0.689

0.666

0.675

0.744

0.736

0.597

0.673

0.220



● Acknowledgments


A1

A2

A3

A4

A5

A6

A7

A8

HHH

0.708

0.683

0.435

0.424

0.522

0.701

0.715

0.853

HHL

0.382

0.735

0.224

0.190

0.510

0.572

0.455

0.777

HLH

0.400

0.451

0.091

0.262

0.442

0.260

0.196

0.516

HLL

0.032

0.267

0.204

0.373

0.498

0.045

0.189

0.294

LHH

0.547

0.732

0.807

0.709

0.652

0.530

0.696

0.729

LHL

0.286

0.341

0.546

0.820

0.702

0.433

0.411

0.328

LLH

0.250

0.320

0.465

0.128

0.579

0.427

0.333

0.494

LLL

0.038

0.380

0.392

0.306

0.114

0.486

0.398

0.052


Thesis Defense

Slide 20 of 36

Discussion

● Overview

n

● Modeling Comparison ● Thesis Model ● Thesis Structural Model ● Thesis Measurement Model ● Thesis Objectives ● Estimation Algorithm ● Evaluation Criteria ● Proficiency Space Studies ● Polytomous Attributes ● Polytomous Attribute Studies ● GLMPM Application ● Future Research Directions ● Acknowledgments

n

Thesis Defense

Item parameter results indicated good recovery in efficacy study, poor recovery in robustness study. u Estimation of π ∗ and r ∗ parameters was generally accurate in efficacy studies. u c parameter was estimated with most difficulty, particularly in cases with high completeness. u In robustness studies, c parameter was nearly always under-estimated. Estimation of the measurement model latent traits varied by trait type. u Classification rates were considerably above chance in efficacy studies. u Classification rate plummeted in robustness studies. u Similar to Hartz (2002), estimation of θ parameter was difficult in almost all cases (although is better in robustness studies). Slide 21 of 36

Discussion (Continued)

● Overview

n

● Modeling Comparison ● Thesis Model ● Thesis Structural Model ● Thesis Measurement Model ● Thesis Objectives

n

● Estimation Algorithm ● Evaluation Criteria ● Proficiency Space Studies ● Polytomous Attributes

n


n

n

Thesis Defense

In efficacy studies, structural model parameters were accurately estimated with a limited amount of bias. GLMPM is accurately estimated across a wide variety of RUM conditions when proficiency space is modeled correctly. Mean structure and measurement model parameter estimates are not robust when correlational structure is not modeled. Severe lack of robustness when correlation is not modeled, indicating that correlation should always be parameterized. Effects of not modeling correlation were not limited to structural model/proficiency space parameters, but were shown to cause extreme bias in c parameter estimates.

Slide 22 of 36

Generalization to Polytomous Attributes


P (Xij = 1|αi , θi , qj ) = Pcj (θi ) πj∗

● Thesis Model ● Thesis Structural Model ● Thesis Measurement Model ● Thesis Objectives ● Estimation Algorithm


j +θi ) e 1+e1.701(cj +θi )

n

For dichotomous attributes (Hartz, 2002): ujk (αik , qjk ) = (1 − αik ) × qjk For polytomous attributes: ujk (αik = l, qjk ) = fjkl × qjk , such that: fjk0 = 1 > fjk1 > . . . > fjkL = 0 Constrained polytomous attribute RUM (single additional parameter per attribute level l): f1kl = f2kl = . . . = fJkl GLMPM is generalized by incorporating ordered κkl parameters for each attribute.

● GLMPM Application ● Future Research Directions

n

n

n

Thesis Defense

k=11.701(c

(αik ,qjk )

With “completeness” term: Pcj (θi ) =


● Acknowledgments

∗u

rjk jk

!

n

● Evaluation Criteria ● Proficiency Space Studies

K Y

Slide 23 of 36

Polytomous Attribute RUM Illustrated 1.0

π*j = 0.90 | r*j1 = 0.30 | r*j2 = 0.10 | fj11 = 0.50 | fj21 = 0.75 | cj = 2.50

0.6 0.4 0.0

0.2

P(X ij =1|α i)

0.8

α = 22 α = 21 α = 20 α = 12 α = 11 α = 10 α = 02 α = 01 α = 00

−4

−2

0

2

4

θi

Thesis Defense

Slide 24 of 36

Polytomous Attribute Studies

● Overview

n

● Modeling Comparison ● Thesis Model ● Thesis Structural Model ● Thesis Measurement Model ● Thesis Objectives ● Estimation Algorithm

n

● Evaluation Criteria ● Proficiency Space Studies ● Polytomous Attributes ● Polytomous Attribute Studies ● GLMPM Application

n


n

Thesis Defense

Test the efficacy of estimation algorithm at recovery of structural (GLMPM) and measurement (Polytomous Attribute RUM) model parameters across varying conditions. Two attributes trichotomous (0, 1, 2), six attributes dichotomous (0, 1). Measurement model (RUM) conditions: u Q-matrix complexity (low and high). u Item parameter cognitive structure (low and high). u Item completeness (low and high). Structural model (GLMPM) condition: u Higher Order Latent Proficiency Space - β = 0.

Slide 25 of 36

Unconstrained Polytomous Attribute Studies Low Complexity, High Cognitive Structure, High Completeness π*

0.8

0.9

1.0

0.6 0.4 0.0

0.2

Estimated 0.0

0.1

0.2

0.3

0.4

0.0

0.1

0.2

0.3

True

True

c

κ

λ

2.0

2.5

3.0

0.5

0.6

0.8

Estimated

0.4

0.6

1.0 0.0 −1.0

−0.5

Estimated

0.5

2.5 2.0 1.5

1.5

0.4

1.0

True

1.0

−1.0

−0.5

0.0 True

G

θ

0.5

1.0

0.4

0.5

0.6

0.7

0.8

0.9

1.0

True

2 −2 −4

−4

−2

0

Estimated

2

4

True

0

Estimated

0.3 0.1 0.0

0.7

4

1.0

Estimated

0.2

Estimated

0.9 0.8 0.7

Estimated

0.6 0.5

0.6

3.0

0.5

−4

−2

0 True

Thesis Defense

f

0.4

1.0

r*

2

4

−4

−2

0

2

4

True

Slide 26 of 36

Unconstrained Polytomous Attribute Studies


Bias Condition

π∗

r∗

c

f

κ

λ

θ

G


HHH

0.004

0.007

-0.271

0.005

-0.033

-0.003

-0.037

-0.013


HHL

-0.003

0.016

0.217

0.038

0.046

-0.012

-0.085

0.020

HLH

0.009

0.010

-0.489

0.106

-0.091

-0.034

-0.300

0.032


HLL

0.000

-0.018

0.306

0.150

-0.101

0.021

0.158

-0.014


LHH

0.006

0.007

-0.191

0.042

-0.007

-0.017

-0.170

0.008

LHL

-0.016

0.010

0.082

-0.026

0.023

-0.008

-0.052

0.026


LLH

-0.005

-0.012

-0.041

0.153

-0.186

-0.035

-0.334

-0.046


LLL

-0.019

0.009

0.151

0.095

-0.035

0.034

0.292

-0.004



● Acknowledgments

MAD π∗

r∗

c

f

κ

λ

θ

G

HHH

0.022

0.030

0.460

0.048

0.044

0.026

0.645

0.860

HHL

0.022

0.041

0.276

0.096

0.060

0.057

0.494

0.390

HLH

0.023

0.059

0.616

0.154

0.173

0.066

0.633

0.438

HLL

0.024

0.074

0.373

0.189

0.182

0.080

0.507

0.436

LHH

0.021

0.026

0.457

0.094

0.050

0.060

0.588

0.430

LHL

0.025

0.031

0.138

0.096

0.065

0.031

0.386

0.385

LLH

0.020

0.038

0.323

0.206

0.194

0.046

0.643

0.424

LLL

0.036

0.079

0.239

0.157

0.162

0.074

0.453

0.476

Condition


Thesis Defense

Slide 27 of 36

Unconstrained Polytomous Attribute Studies


Classification Rate

● Thesis Model

Condition


Whole

A1

A2

A3

A4

A5

A6

A7

A8

Average

Pattern

HHH

0.767

0.803

0.887

0.864

0.928

0.915

0.934

0.956

0.882

0.439

HHL

0.704

0.794

0.868

0.858

0.880

0.888

0.891

0.925

0.851

0.374


HLH

0.572

0.680

0.864

0.818

0.802

0.855

0.864

0.904

0.795

0.202


HLL

0.622

0.541

0.813

0.818

0.882

0.850

0.836

0.848

0.776

0.194


LHH

0.770

0.808

0.950

0.945

0.933

0.926

0.960

0.906

0.900

0.444


LHL

0.666

0.720

0.889

0.901

0.919

0.878

0.926

0.899

0.850

0.331


LLH

0.618

0.564

0.891

0.878

0.938

0.930

0.870

0.892

0.823

0.230


LLL

0.569

0.486

0.780

0.819

0.836

0.840

0.820

0.820

0.746

0.127



● Acknowledgments


A1

A2

A3

A4

A5

A6

A7

A8

HHH

0.631

0.690

0.763

0.722

0.857

0.826

0.859

0.909

HHL

0.549

0.683

0.737

0.718

0.754

0.766

0.768

0.847

HLH

0.358

0.502

0.702

0.621

0.602

0.705

0.724

0.800

HLL

0.422

0.346

0.602

0.620

0.749

0.693

0.669

0.673

LHH

0.648

0.659

0.900

0.890

0.866

0.846

0.917

0.808

LHL

0.473

0.556

0.773

0.792

0.838

0.752

0.851

0.794

LLH

0.412

0.372

0.782

0.742

0.871

0.851

0.703

0.785

LLL

0.347

0.268

0.560

0.639

0.648

0.670

0.604

0.593


Thesis Defense

Slide 28 of 36

Discussion

● Overview

n


n

n

Thesis Defense

Item parameter results indicated generally good recovery of true parameters. u Estimation of π ∗ and r ∗ parameters was accurate. u c parameter was estimated with most difficulty, particularly in cases with high completeness. u f parameter recovery was generally accurate for both constrained and unconstrained models. Estimation of the measurement model latent traits varied by trait type. u Classification rates were typically well above chance, although reduced for polytomous attributes. u Estimation of θ parameter was difficult in almost all conditions. Structural model parameters were accurately estimated with a limited amount of bias.

Slide 29 of 36

Illustrative Application

● Overview

n


Fraction Subtraction data set. u 20 item Math test given to 2,144 middle school students. u Q-matrix has eight attributes (average 2.75 attributes per item):


1. Convert a whole number to a fraction.


2. Separate a whole number from fraction.


3. Simplify before subtracting.


4. Find a common denominator.

● GLMPM Application ● Future Research Directions

5. Borrow from whole number part.

● Acknowledgments

6. Column borrow to subtract the second numerator from the first. 7. Subtract numerators. 8. Reduce answers to simplest form.

n

n

Thesis Defense

Dichotomous attribute, constrained polytomous attribute, and unconstrained polytomous attribute models estimated. u In polytomous attribute analysis, all attributes were treated as trichotomous (0, 1, 2). Higher order proficiency space structural model.

Slide 30 of 36

Fraction Subtraction Analysis

● Overview ● Modeling Comparison ● Thesis Model ● Thesis Structural Model ● Thesis Measurement Model ● Thesis Objectives ● Estimation Algorithm ● Evaluation Criteria ● Proficiency Space Studies ● Polytomous Attributes ● Polytomous Attribute Studies ● GLMPM Application ● Future Research Directions ● Acknowledgments

Thesis Defense

n

Measurement model parameter estimates: u π ∗ and r ∗ parameter estimates were generally indicative of quality items and were consistent across models. u c parameter estimates were generally high (with few exceptions), and became higher as additional attribute levels were added. u In the constrained polytomous model, four polytomous item parameters (f ) approached zero, indicating a questionable application of polytomous attributes. u A similar phenomenon was found with many of the unconstrained polytomous model parameters approaching zero.

Slide 31 of 36



Structural model parameter estimates: Dichotomous

● Thesis Model ● Thesis Structural Model ● Thesis Measurement Model ● Thesis Objectives

Attribute


Constrained

Unconstrained

Polytomous

Polytomous

κk1

λ

κk1

κk2

λ

κk1

κk2

λ

1

-0.053

0.957

-0.292

-0.023

0.963

-0.057

0.705

0.966

2

-1.175

0.699

-1.153

0.163

0.753

-1.070

0.576

0.791


3

-0.558

0.608

-3.279

-2.315

0.354

-3.446

-2.107

-0.424


4

-0.221

0.912

-0.266

-0.126

0.917

-0.298

-0.188

0.897

5

0.022

0.892

-0.140

0.085

0.892

-0.009

0.050

0.900

6

-0.506

0.989

-0.680

2.982

0.986

-2.711

-1.537

0.296

7

-1.009

0.969

-1.026

-0.392

0.962

-1.021

-0.386

0.994

8

0.104

0.916

-2.534

-0.030

0.839

-0.271

0.347

0.874

θ

-

0.819

-

-

0.705

-

-

0.714

● Evaluation Criteria ● Proficiency Space Studies


Thesis Defense

Slide 32 of 36



Model fit indices:

● Thesis Model

Analysis

● Thesis Structural Model ● Thesis Measurement Model ● Thesis Objectives ● Estimation Algorithm

Parameters

AIC

BIC

log(L)

Dichotomous

113

117,766.0

117,555.3

-58,770.0

Constrained Polytomous

129

112,382.2

112,139.5

-56,062.1

Unconstrained Polytomous

185

115,556.6

115,201.9

-57,593.3

● Evaluation Criteria ● Proficiency Space Studies ● Polytomous Attributes

n


Thesis Defense

n

Fit indices calculated by Monte Carlo approximation. Constrained polytomous model is selected by both AIC and BIC.

Slide 33 of 36

Fraction Subtraction Discussion

● Overview

n


n

Thesis Defense

Although AIC and BIC suggest otherwise, all signs point to the dichotomous attribute model as providing most intuitive explanation of parameter estimates. u Minimal changes in π ∗ , r ∗ , and c parameter estimates. u Polytomous attribute structural model κ parameters indicating some 99% of distribution concentrated on single attribute level. u Negative λ parameter in unconstrained polytomous attribute structural model. Dichotomous models have shown to be reasonable in other applications using the DINA model (de la Torre and Douglas, in press). u If DINA model fits adequately, even dichotomous RUM may be superfluous.

Slide 34 of 36

An Eye to the Future

● Overview

n


n

● Evaluation Criteria ● Proficiency Space Studies ● Polytomous Attributes ● Polytomous Attribute Studies ● GLMPM Application ● Future Research Directions ● Acknowledgments

n n

Thesis Defense

Evaluation of appropriateness of cognitive diagnosis models in psychological testing (such as personality assessment or clinical diagnosis). Generalization of structural model for latent trait associations. u General model for attribute (tetrachoric) correlations. u Appropriateness of structural equation model in applications. Better indices of model fit and comparison. Modifications of polytomous attribute model. u Omission of Pc (θ) term. u Parameterization of Pc (θ) as discrete polytomous attribute with Q-matrix entries on every item. u Application with other cognitive diagnosis models (such as NIDA). Slide 35 of 36

Acknowledgments

● Overview

n

● Modeling Comparison ● Thesis Model ● Thesis Structural Model ● Thesis Measurement Model ● Thesis Objectives ● Estimation Algorithm ● Evaluation Criteria

n n n

● Proficiency Space Studies ● Polytomous Attributes ● Polytomous Attribute Studies ● GLMPM Application ● Future Research Directions ● Acknowledgments

n n n n

Thesis Defense

Jeff Douglas Bill Stout Larry Hubert David Budescu Carolyn Anderson Louis Roussos Bob Henson Sara Templin

Slide 36 of 36