Marsha Lynn Schroeder, 1986 .... I am also indebted to Todd Rogers for introducing me to ... thank Todd for his careful reading of and thoughtful comments on ...
INFERENTIAL PROCEDURES FOR MULTIFACETED COEFFICIENTS OF GENERALIZABILITY by MARSHA LYNN A.,
The U n i v e r s i t y
SCHROEDER
of B r i t i s h
Columbia,
1982
THESIS SUBMITTED IN PARTIAL FULFILMENT OF THE
REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY in
THE
FACULTY OF GRADUATE STUDIES Department
We a c c e p t to
THE
this
of Psychology
thesis
the required
as conforming standard
UNIVERSITY OF BRITISH COLUMBIA September
1986
© M a r s h a Lynn S c h r o e d e r ,
1986
In p r e s e n t i n g
this thesis
i n partial
fulfilment of the
r e q u i r e m e n t s f o r an a d v a n c e d d e g r e e a t t h e U n i v e r s i t y of B r i t i s h it
Columbia, I agree that
freely available
agree that
f o rreference
permission
the Library
shall
and study.
I
make
further
f o r extensive copying o f t h i s
thesis
f o r s c h o l a r l y p u r p o s e s may be g r a n t e d by t h e h e a d o f my d e p a r t m e n t o r by h i s o r h e r r e p r e s e n t a t i v e s . understood that for
copying or p u b l i c a t i o n
f i n a n c i a l gain
PSYCHOLOGY
The U n i v e r s i t y o f B r i t i s h 1956 Main Mall V a n c o u v e r , Canada V6T 1Y3 Date
DE-6
C3/81)
S
E
P
T
«
2 9
>
1
9
8
thesis
s h a l l n o t be a l l o w e d w i t h o u t my
permission.
Department o f
of this
It is
6
Columbia
written
INFERENTIAL PROCEDURES FOR MULTIFACETED COEFFICIENTS OF GENERALIZABILITY
ABSTRACT G e n e r a l i z a b i l i t y t h e o r y was d e v e l o p e d by Cronbach as an a l t e r n a t i v e to c l a s s i c a l t e s t score r e l i a b i l i t y theory. G e n e r a l i z a b i l i t y uses an e x p e r i m e n t a l d e s i g n a p p r o a c h t o r e l i a b i l i t y t h a t p e r m i t s t h e systematic e v a l u a t i o n o f s e v e r a l sources o f e r r o r s i m u l t a n e o u s l y . The c o e f f i c i e n t o f g e n e r a l i z a b i l i t y (CG) i s a s i n g l e number summarizing t h e d e p e n d a b i l i t y o f t h e measurement p r o c e s s . In t h e p r e s e n t s t u d y a n o r m a l i z i n g t r a n s f o r m a t i o n was f i r s t a p p l i e d t o a f u n c t i o n o f t h e CG. The d e l t a method was a p p l i e d t o t h e t r a n s f o r m e d CGs f o r f o u r d i f f e r e n t t w o - f a c e t e x p e r i m e n t a l d e s i g n models t o d e v e l o p a s y m p t o t i c v a r i a n c e e x p r e s s i o n s f o r t h e CGs. The a c c u r a c y o f t h e v a r i a n c e e x p r e s s i o n s was t e s t e d v i a Monte C a r l o simulations. I n .these s i m u l a t i o n s t h e Type I e r r o r c o n t r o l was investigated. The m a j o r i t y o f t h e s i m u l a t i o n s were c o n d u c t e d u s i n g a t w o - f a c e t f u l l y random e x p e r i m e n t a l d e s i g n , c o r r e s p o n d i n g t o a three-way random e f f e c t s a n a l y s i s o f v a r i a n c e . A t o t a l o f 81 c o m b i n a t i o n s o f sample s i z e , f a c e t c o n d i t i o n s , and p o p u l a t i o n CG v a l u e s were i n v e s t i g a t e d . The r e s u l t s s u g g e s t e d t h a t the p r o c e d u r e g e n e r a l l y was p r e c i s e i n i t s c o n t r o l o f Type I e r r o r . The r e s u l t s w e r e so^twhat l e s s p r e c i s e when o n l y two f a c e t c o n d i t i o n s were sampled. F i v e o t h e r s i d e s t u d i e s were c o n d u c t e d . T h r e e o f t h e s e used o t h e r t w o - f a c e t models: a d e s i g n w i t h one f i x e d f a c e t , a d e s i g n w i t h a f i n i t e f a c e t , and a d e s i g n w i t h a n e s t e d f a c e t . The r e s u l t s o f t h e s e s t u d i e s were s i m i l a r t o those found i n t h e l a r g e r s t u d y ; g e n e r a l l y good Type I e r r o r c o n t r o l was r e a l i z e d . An a d d i t i o n a l s t u d y l o o k e d a t t h e performance o f t h e v a r i a n c e e x p r e s s i o n i n t h e p r e s e n c e o f n e g a t i v e v a r i a n c e component e s t i m a t e s . Results i n t h i s s e c t i o n o f t h e s t u d y s u g g e s t e d t h a t s u c h n e g a t i v e component e s t i m a t e s d i d n o t a d v e r s e l y a f f e c t Type I e r r o r c o n t r o l . The f i n a l study i n v e s t i g a t e d the performance of the v a r i a n c e e x p r e s s i o n with dichotomous d a t a . The r e s u l t s i n d i c a t e d t h a t Type I e r r o r c o n t r o l was n o t as p r e c i s e w i t h two f a c e t c o n d i t i o n s as i t was w i t h f i v e o r e i g h t c o n d i t i o n s . In t h e s e l a t t e r c a s e s good e r r o r c o n t r o l was realized.
iv Table
of
Contents Page
Abstract
i i
List
of T a b l e s
vi
List
of F i g u r e s
ix
Acknowledgements
x
Chapter
1
1 - Introduction
The O b j e c t
3
o f Measurement
Universes
3
Components o f V a r i a n c e
4
G and D S t u d i e s
6
Two
Kinds
of D e c i s i o n s ; Two
The C o e f f i c i e n t Estimating
Chapter
of E r r o r
Variance
of G e n e r a l i z a b i l i t y
12
Research
of t h e P r e s e n t
2 - Mathematical
7 9
the C G
Generalizability Purpose
Kinds
13
and A p p l i c a t i o n
18
Study
Development
The A p p r o x i m a t e D i s t r i b u t i o n
20
of t h e CG
The N o r m a l i z a t i o n of a F u n c t i o n
20
of t h e CG
25
The D e r i v a t i o n of an Asymptotic Variance for
Variance
Expression
f o r t h e CG
26
Expressions CGs A r i s i n g
from
Other
Experimental
Designs
29
Chapter
3 - Method
36
Data G e n e r a t i o n Overview
36
of t h e S t u d y
for Design Three Other
VII with
B o t h F a c e t s Random
Models
39
A Special
Condition:
A Special
C o n d i t i o n : Dichotomous Data
Testing Chapter
Zero V a r i a n c e
Expression
VII with F i x e d
Design
VII with F i n i t e
Design
V-B w i t h
Design
44 49
Design
Treatment
41 43
4 - Results VII with
Chapter
Components
t h e Adequacy o f t h e V a r i a n c e
Design
The
38
B o t h F a c e t s Random
49
Item F a c e t Random R a t e r
57 Facet
B o t h F a c e t s Random
of N e g a t i v e
Variance
57 60
Components
VII w i t h Dichotomous Data
61 62
5 - C o n c l u s i o n s and a Worked Example o f t h e P r o c e d u r e s
Some O b s e r v a t i o n s Implications for
about
the E m p i r i c a l
67 Results
67
of t h e Study
t h e Use o f G e n e r a l i z a b i l i t y
Limitations
of the P r e s e n t
Suggestions
f o r Future
A Worked Example U s i n g
Theory
Study
72
Research the Present
70
73 Procedure
75
References
77
Appendix
84
A
vi
List
of T a b l e s Page
Table
1
Expected both
Table
2
Mean S q u a r e s f o r D e s i g n
F a c e t s Random
3
Components
4
R e s u l t s and E s t i m a t e d f o r Psychopathy Data
Summary o f D e s i g n s a n d C o n d i t i o n s Empirical
Table
22
A n a l y s i s of V a r i a n c e Variance
Table
Proportion
o f Type
(Design V I I , both 5
I Errors for p
2
Estimates P o i n t s of
and A c t u a l
o f Type I E r r o r s f o r p
(Design V I I , both
2
52
Estimates
below S e l e c t e d P e r c e n t i l e
t h e U n i t Normal D i s t r i b u t i o n
P o i n t s of
and A c t u a l
o f Type I E r r o r s f o r p
(Design V I I , both
= .70
F a c e t s Random)
P r o p o r t i o n of the Standardized
Proportion
= .50 51
below S e l e c t e d P e r c e n t i l e
Proportion
Falling
P o i n t s of
F a c e t s Random)
t h e U n i t Normal D i s t r i b u t i o n
6
Estimates
and A c t u a l
P r o p o r t i o n of the S t a n d a r d i z e d Falling
Table
45
below S e l e c t e d P e r c e n t i l e
the U n i t Normal D i s t r i b u t i o n
23
f o r the
Investigation
P r o p o r t i o n of the S t a n d a r d i z e d Falling
Table
VII with
F a c e t s Random)
2
= .90 53
V 1 1
Table
7
Mean V a l u e s
and O v e r a l l
Parameter V a l u e s
and F a c e t
(Design V I I , both Selected
Chi-square
V a l u e s by
Conditions
F a c e t s Random) f o r
Percentile
Normal D i s t r i b u t i o n
P o i n t s of the U n i t a n d T h r e e L e v e l s of Type
I Error Table
8
54
P r o p o r t i o n of the S t a n d a r d i z e d Falling
below S e l e c t e d P e r c e n t i l e
t h e U n i t Normal D i s t r i b u t i o n
Table
9
Proportion
o f Type
Item F a c e t
Fixed)
I E r r o r s (Design V I I , 58
t h e U n i t Normal D i s t r i b u t i o n
Rater Table
10
Facet
of Type
I E r r o r s (Design V I I ,
Finite)
59
t h e U n i t Normal D i s t r i b u t i o n
both
Estimates
below S e l e c t e d P e r c e n t i l e
Proportion
P o i n t s of
and A c t u a l
P r o p o r t i o n of the S t a n d a r d i z e d Falling
Estimates
below S e l e c t e d P e r c e n t i l e
Proportion
P o i n t s of
and A c t u a l
P r o p o r t i o n of the S t a n d a r d i z e d Falling
Estimates
o f Type
F a c e t s Random)
P o i n t s of
and A c t u a l
I E r r o r s ( D e s i g n V-B, 61
vi i i Table
11
P r o p o r t i o n of the S t a n d a r d i z e d Falling
below S e l e c t e d P e r c e n t i l e
the U n i t Normal D i s t r i b u t i o n Proportion
Estimates 12
and A c t u a l
Variance
Proportion
P o i n t s of
and A c t u a l
o f Type I E r r o r s f o r D i c h o t o m i z e d
Data and Comparable C o n d i t i o n s Continuous
63
Estimates
below S e l e c t e d P e r c e n t i l e
the U n i t Normal D i s t r i b u t i o n
Random)
Component
( D e s i g n V I I , B o t h F a c e t s Random)
P r o p o r t i o n of the S t a n d a r d i z e d Falling
P o i n t s of
o f Type I E r r o r s f o r Two
Treatments of Negative
Table
Estimates
Data
with
(Design V I I , both
Facets 66
List
of
Figures Page
Figure
1
Layout
of
Variance
Data
for
Design
VII
Analysis
of 37
x
\
Acknowledgements
I would support project.
like
and encouragement I would
consenting
thank Todd
me
like
committee. instrumental would text
like
I also
reading
drafts
wish t o express
f o r h i s many h e l p f u l
indebted
for his careful
t o thank
t o Todd
o f and
of t h i s t h e s i s .
assistance
comments
Rogers f o r I also
wish t o
thoughtful I especially
Ralph Hakstian f o r c h a i r i n g
His patient
for their
Papageorgis f o r
to G e n e r a l i z a b i l i t y theory;
my
and encouragement
were
t o the c o m p l e t i o n of t h i s t h e s i s . F i n a l l y , I t o thank
preparation;
considerable
t o thank D e m e t r i
I am a l s o
comments on p r e v i o u s would
committee
throughout t h e c o u r s e of t h i s
to Jim Steiger
suggestions.
introducing
like
thesis
t o s e r v e on my c o m m i t t e e .
my a p p r e c i a t i o n and
t o thank my
Klaus Schroeder f o r h i s a s s i s t a n c e
I would
also
like
support throughout t h i s
with
t o thank K l a u s f o r h i s project.
1 Chapter
1
Introduction Generalizability (Cronbach,
theory
1963;
Rajaratnam, Cronbach, & G l e s e r , of c l a s s i c a l t e s t
Classical
score
parallel this
test
by
1972;
theory
1965)
score
as
i s b a s e d on
i s regarded
generalizability
theory
as an
Rajaratnam, a theory.
the a s s u m p t i o n theory
r e s t r i c t i v e a s s u m p t i o n . Measurement model
Cronbach,
reliability
measurements; g e n e r a l i z a b i l i t y
classical
Cronbach
G l e s e r , Cronbach, &
liberalization
on
proposed
G l e s e r , Nanda, & R a j a r a t n a m ,
Rajaratnam, & G l e s e r , 1965;
was
of
does not
rely
error within
the
amorphous q u a n t i t y ; i n
measurement e r r o r i s examined
systematically.
The has
view of measurement
been q u e s t i o n e d
1947,
1970;
sources
may
not
an
(e.g.,
next
presentation,
with
e v a l u a t i o n of design
the
same form of an when p a r a l l e l
theory
this
theory
be
different arising
instrument forms of
permits
thesis
will
introduce
p e r t a i n i n g to
developed.
the
the
of e r r o r by
i n some d e t a i l . F o l l o w i n g
will
Cronbach,
t o measurement.
i n f e r e n t i a l procedures
generalizability
theory
s e v e r a l sources
approach
s e c t i o n s of
generalizability
from
e q u i v a l e n t ; f o r example, e r r o r
used. G e n e r a l i z a b i l i t y
experimental The
a number of w r i t e r s
from e r r o r a r i s i n g
are
simultaneous
be
testing
differs
instrument
amorphous q u a n t i t y
G u l l i k s e n , 1936). E r r o r s a r i s i n g
from r e p e a t e d likely
by
e r r o r as an
this
using
2 To
i n t r o d u c e the
n o t i o n of an
experimental
a p p r o a c h t o measurement, c o n s i d e r t h e taken In
from
this
a study
study
checklist
used
assessed.
The
the
by
Schroeder,
checklist
of p s y c h o p a t h y .
trained
raters
who
f o l l o w s we year.
on
will
the
be
22
f i v e years
items
by
two
with
were a s s e s s e d
calculated.
consistency
f o r each
agreement, the the means and of
the
For
ratings
rater,
significance
over
ANOVA. The
calculate
seven
the
interrater
the
(i.e.,
22
generalizability
they
theory;
sound d a t a
scores
guide
on sum
item
generalizability data
as a
three-way
(ANOVA) and
mean s q u a r e s
key
the
individual
i n m a t e s ) X Items X
the
raters
(i.e.,
s e p a r a t e v a r i a n c e component are
of
between
i t e m s ) , and
resulting
These v a r i a n c e e s t i m a t e s
design
reliability
of
of v a r i a n c e
fully-crossed
test
a coefficient
would c o n c e p t u a l i z e t h e s e
Persons
single
internal
theory,
corresponding
inmates
of
evaluated. Using
analysis
item to a
index
c o u l d be
random e f f e c t s
or more
classical
of
of d i f f e r e n c e s
was
In what
for a
indexes
statistics we
two
a number of
raters.
variances for t o t a l test given
of a
tapping
by
u s i n g the
example an
(1983).
inmates
items
the data
s c o r e m o d e l , a number of d i f f e r e n t would be
Hare
t h e a p p l i c a b i l i t y of e a c h
concerned
If r e l i a b i l i t y
22
I t i s t y p i c a l l y used
i n m a t e . In e a c h of
were e v a l u a t e d
and
in prison
i s composed of
judge
example
(reliability)
t o measure p s y c h o p a t h y
aspects
particular
following
Schroeder,
generalizability
design
conduct
the
Raters are used
estimates.
e l e m e n t s of the
researcher
c o l l e c t i o n procedures.
to
i n the
They a r e
also
3 u s e d t o compute
the value
generalizability reliability
At
( C G ) , a s i n g l e number s u m m a r i z i n g t h e
or d e p e n d a b i l i t y
this
terminology
of the c o e f f i c i e n t of
point
o f t h e measurement
some g e n e r a l i z a b i l i t y
are presented
to help
process.
theory
clarify
c o n c e p t s and
the material
that
follows. The O b j e c t
o f Measurement
The o b j e c t the
element
o f measurement
in generalizability
o f t h e s t u d y a b o u t w h i c h one w i s h e s t o make
judgments. In t h e S c h r o e d e r of measurement
were
e t a l . (1983) s t u d y
inmates. Throughout
this
objects
o f measurement
will
objects
o f measurement
in a generalizability
could
be a n y p o p u l a t i o n
evaluated. paintings
theory i s
the objects
t h e s i s the
be r e f e r r e d t o a s p e r s o n s . The study,
of organisms or o b j e c t s ,
F o r example, t h e o b j e c t s
however,
t o be
o f measurement
c o u l d be
or animals.
Universes The e m p h a s i s generalizability conditions
used
i n both c l a s s i c a l theory
i n the study
Any o b s e r v a t i o n Facets
i s placed
test
score
t h e o r y and
on g e n e r a l i z i n g b e y o n d t h e
to a larger set of c o n d i t i o n s .
i s made under a s e t o f c o n d i t i o n s .
a r e composed o f conditions
(the term
a n a l o g o u s t o t h e ANOVA
t e r m factor;
level).
c a n be c l a s s i f i e d
An o b s e r v a t i o n
facet i s
the term c o n d i t i o n t o according
t o the
4 conditions universe
of
conditions in
o f t h e f a c e t s under w h i c h admissible
consists
of a f a c e t that t h e o r e t i c a l l y The universe
a study.
conditions
observations
i t was made. The
of
generali
of t h e f a c e t over
could
universe
the universe
of a d m i s s i b l e
w h i c h one w i s h e s t o g e n e r a l i z e .
would
like
study.
Facets
When a f a c e t admissible facet
study,
f a c e t s over
which t h e
to generalize are i d e n t i f i e d . f a c e t a r e sampled
c a n be f i x e d ,
i s fixed,
for inclusion in
random, o r f i n i t e
are included
effects.
i n the study.
When a
i s random, a number o f c o n d i t i o n s a r e sampled infinite
o b s e r v a t i o n s . When a f i n i t e
universe
universe
of admissible
Related universe
facet i s included
i n a study, a
t o the concept
notion
finite
observations. of u n i v e r s e s
s c o r e . The u n i v e r s e
score
randomly
of a d m i s s i b l e
number o f c o n d i t i o n s a r e sampled a t random from a
test
Two o r
a l l c o n d i t i o n s of the u n i v e r s e of
observations
from t h e t h e o r e t i c a l l y
of the
observations.
more c o n d i t i o n s o f e a c h the
i n some
of g e n e r a l i z a t i o n i s a subset
In a g e n e r a l i z a b i l i t y researcher
be i n c l u d e d
zat i on r e p r e s e n t s a l l
These two terms may be synonymous; however, situations
of a l l
of true
score
i s t h e concept of
i s like
the c l a s s i c a l
score.
Components o f V a r i a n c e The
variance
generalizability containing
data
components a r e t h e b u i l d i n g b l o c k s o f theory.
Consider,
(persons'
scores
f o r example, a m a t r i x
on i t e m s ) o b t a i n e d
from t h e
5 administration
of a t e s t .
generalizability object even the
of measurement, p e r s o n s ,
corresponding
the
where e a c h p e r s o n
a
score
+ a l
2
classical items,
test
a ,
heterogeneity difficulty
Item
score
reflects
2
residual
variance
variance,
t h e o b j e c t s of score
The v a r i a n c e
variance in
associated
desirability.
which
i s composed
levels
The t e r m
of
p l u s e r r o r due t o f a c e t s o t h e r
considered
fully-crossed
f a c e t s and random
two-facet
item
item
a . pi, e 2
with
i s the
of t h e two-way P e r s o n
The S c h r o e d e r e t a l . (1983) d e s i g n raters
i s variance
d i f f e r e n c e s among t h e i t e m s ;
or s o c i a l
interaction
can be
components:
true
can i n d i c a t e d i f f e r i n g
variance
explicitly
theory.
as a
2
score
is like
on
+ a . pi,e
2
due t o d i f f e r e n c e s among r e s p o n d e n t s , this
(or i s scored,
variance
variance
2
one-facet
Items i s c o n s i d e r e d
observed
t h i s model a , t h e u n i v e r s e P
measurement;
a facet
In a b a s i c
rates him/herself
(X . ) = a pi p
2
The
i t i s i n c l u d e d as a f a c t o r i n
as t h e sum of t h r e e
(1 )
i s not c o n s i d e r e d
and where
f a c e t ) the t o t a l
expressed
(Items) d e s i g n .
a n a l y s i s of v a r i a n c e .
same s e t o f i t e m s
random
i s described in
as a o n e - f a c e t
t h o u g h , a s shown n e x t ,
design
In
theory
This design
design.
than
those
fluctuation. i s an items For t h i s
by
design
the
by
6
total
observed
s c o r e v a r i a n c e c a n be e x p r e s s e d a s :
a (X
. ) = a + a pir p 1
2
+ a
2
2
2
r
+
a.
+ a pr 2
2
pi
(2) + a?
+a . 2
ir
The v a r i a n c e components estimated
u s i n g from
t h e ANOVA e x p e c t e d
pir,e
i n equations
statistical
(1) a n d (2)
theory
estimates of the expected
substituting
the v a l u e s
for a a
2
P
2
for
i nthe
estimates of the
F o r example, t h e
i s g i v e n by = (MS
the two-facet
expressions the
P
mean s q u a r e s ,
c a n be c a l c u l a t e d .
squares
mean s q u a r e s . By
f o r t h e mean s q u a r e s
f o r the expected
v a r i a n c e components estimate
the e x p r e s s i o n s f o r
mean s q u a r e s . The o b t a i n e d mean
are unbiased
equations
are
P
- MS
random
. - MS + MS . ) Pi pr pir,e
effects
f o r the expected
facets are fixed,
mean s q u a r e s
random,
(1967) p r o v i d e d r u l e s
m o d e l . The form
or f i n i t e .
for writing
of the
d e p e n d s on w h e t h e r Millman
and G l a s s
these expressions
(also
see C r o n b a c h e t a l . , 1972, Ch. 2 ) . G and D S t u d i e s C r o n b a c h e t a l . (1972) d i s t i n g u i s h e d of
study.
conducted be u s e d
The f i r s t ,
the g e n e r a l i z a b i l i t y
between
two t y p e s
or G study, i s
t o o b t a i n e s t i m a t e s o f t h e v a r i a n c e components t o
t o p l a n the second,
purpose of the D study
the d e c i s i o n
o r D s t u d y . The
i s t o make d e c i s i o n s a b o u t
the object
7 of measurement. The D s t u d y provided
i n the G study
Cronbach e t a l . viewed distinct. be
to design
data
u s u a l l y a r e used
Two K i n d s
score
other
the D study
estimate
at
least
estimates.
f o r both
t h e G and D s t u d i e s . of E r r o r
one o f two k i n d s
Variance
of d e c i s i o n i s made. An
to a c r i t e r i o n
e t a l . (1983) s t u d y
universe
or c u t t i n g s c o r e .
i f an inmate were
a s a p s y c h o p a t h when he r e c e i v e d a mean r a t i n g of
2.8 o u t o f a maximum
no d i r e c t
o f 3, an a b s o l u t e
decision
rank o r d e r e d
according
The
standing
relative
decision i n the sense
c o m p a r i s o n s among t h e i n m a t e s t e s t e d were
made. A relative
a d e c i s i o n about Schroeder
being
however, t h e same
would have been made. The d e c i s i o n i s a b s o l u t e that
should
components
s t u d i e s . In p r a c t i c e ,
i s compared
ideally
l a r g e sample G s t u d i e s
i s made when an i n d i v i d u a l ' s
In t h e S c h r o e d e r classified
component
information
t h e G and D s t u d i e s a s b e i n g
o f D e c i s i o n ; Two K i n d s
decision
using
the r e s u l t a n t variance
used
absolute
variance
They b e l i e v e d t h a t
conducted with
In
i s planned
i s made when t h e i n d i v i d u a l s a r e to their
estimated
o f an i n d i v i d u a l
universe
i s then
her or h i s f u t u r e treatment.
e t a l . study
i f the highest
sample had been d e s i g n a t e d
score.
used
t o make
In t h e
s c o r i n g 20% o f t h e
psychopaths, a r e l a t i v e
decision
would have been made.
Just estimates universe
a s t h e two k i n d s
of d e c i s i o n d i f f e r ,
of e r r o r v a r i a n c e score
upon w h i c h
a s s o c i a t e d with
so t o o do t h e
t h e e s t i m a t e s of
the d e c i s i o n s a r e based.
In t h e
8 case
of a b s o l u t e
sources
decisions, error variance
of v a r i a n c e
except
the source
arises
due t o t h e o b j e c t of
measurement. C r o n b a c h e t a l . (1972) l a b e l l e d variance by
expression
some s o u r c e s
e t a l . (1983) s t u d y ,
heterogeneity. across
interaction variance random,
source
a l l inmates
i s accounted
labelled other
do n o t e n t e r
due t o Items
f o r example,
of v a r i a n c e
into
reflects
i s considered
( p o s s i b l e item-inmate
f o r i n the item-person
interaction
c o m p o n e n t ) . When a l l f a c e t s of t h e d e s i g n a r e a (6) incorporates a l l variance
case
with
the object
of a d e s i g n
interaction
interactions
of measurement
between t h e o b j e c t
both
kinds
component
conditions within
of e r r o r v a r i a n c e ,
a fixed
the object
estimate
and a
finite
reflects
upon w h i c h t h e the f a c t
i s the average v a l u e
to using
length
the
that the
of t h e
t h e o b j e c t o f measurement.
i s equivalent for test
due t o
number of
of measurement
score
within
facet
each c o n s t i t u e n t
i s d i v i d e d by t h e t o t a l
universe
correction
with
error.
i s based. T h i s procedure
procedure
facet,
o f measurement
variance
observations
In c o n t r a s t , i n
t o be e r r o r v a r i a n c e . V a r i a n c e
i s considered
variance
of measurement.
incorporating a fixed
of t h e o b j e c t
not c o n s i d e r e d
For
components i n v o l v i n g
2
interaction
facet
This
error
of v a r i a n c e
f o r the e r r o r v a r i a n c e . Variance
the Schroeder
constant
this
decisions,
t h a t due t o t h e o b j e c t of measurement
item
is
of r e l a t i v e
2
the
the
In t h e c a s e
2
Cronbach e t a l . as a ( 6 ) ,
than
in
a (A).
from a l l
This
Spearman-Brown
( C r o n b a c h e t a l . , 1972, p. 8 2 ) .
9 In t h e S c h r o e d e r estimate items
et a l . study
i s the average
f o r e a c h o f two In t h e p r e s e n t
error
variance
variance error is
value
taken
r e s e a r c h , emphasis
The
to the c l a s s i c a l
than
of
i n the study
the
score
n o t i o n of
of i n d i v i d u a l d i f f e r e n c e s
a r e made. (CG)
of g e n e r a l i z a b i l i t y ,
correlation
coefficient
procedure
The i n t r a c l a s s
correlation
test
2
of G e n e r a l i z a b i l i t y
t h e measurement
decisions.
i s p l a c e d on t h e
i s the a ( A ) e r r o r v a r i a n c e . T h i s e r r o r
coefficient
intraclass
44 o b s e r v a t i o n s (22
2
a l s o of r e l e v a n c e
Coefficient
over
CG, i s an
summarizing
i n the case
correlation
t h e adequacy
of r e l a t i v e
coefficient
( C r o n b a c h e t a l . , 1972, p. 17; S h r o u t
Fleiss,
1979). A n e g a t i v e l y b i a s e d b u t c o n s i s t e n t
(Lahey,
Downey, & S a a l ,
lower
correlation
limit
measurement
1983), t h e maximum coefficient
i s zero. Although i s o f t e n used
standardized
in
value
&
statistic of the
i s one; t h e t h e o r e t i c a l
the standard
t o summarize
measurement, t h e CG p r e s e n t s
As
i s the
between e x c h a n g e a b l e measurements o b t a i n e d on
same o b j e c t
intraclass
score
raters).
where c o m p a r i s o n s among p e r s o n s The
inmate's u n i v e r s e
o ( 6 ) . T h i s c o n c e p t u a l i z a t i o n of e r r o r
i s closer
variance
each
e r r o r of
the adequacy of
the advantage of having a
metric.
an example, c o n s i d e r a g a i n
w h i c h a number o f p e r s o n s
number o f i t e m s
measuring
a simple
one-facet
are t e s t e d or evaluated
some a t t r i b u t e .
Further,
design on a
suppose
10
that
the
n.
items
theoretically (i.e.,
each
were sampled a t random from
infinitely
individual
These data
would be
ANOVA w i t h
Persons
to for
of
the
the
as t h e Between
between c l a s s
(Haggard,
[o
p.
items).
effects
i n the d e s i g n .
(here person
1958,
Haggard's treatment attention
observation
as
the
Cronbach et a l . ' s treatment
coefficient
facets;
(i.e.,
t h e CG) p
2
= a/ P 2
of a s i n g l e
coefficient, aggregated
p , 2
items.
i :p
The
i s the
variance)
4 ) . The
expression
],
to Persons, within of t h e
2
intraclass
on
focuses
here
o
and
the
i :p
i s the
Persons.
the
correlation
individual
( h i s treatment
psychometric on
item of
[o
2
P
+
scores averaged
intraclass
o
2
r e p r e s e n t s the
the
average
Cronbach et a l . ' s
reliability
of
over
correlation
/n.]. i:p i-
r e p r e s e n t s the
item while
or
applications).
is
T h u s , H a g g a r d ' s c o e f f i c i e n t , R, reliability
2
of a n a l y s i s
encompasses more t h a n
t h e c o n d i t i o n s of
P
i s focused
unit
a
+
2
2
coefficient
topic
s e t of
population value i s :
i s the v a r i a n c e due P v a r i a n c e due t o Items n e s t e d In
individual
for t h i s design
variance
2
a
factor
coefficient
R = a/ P where
f o r each
a n a l y s e d as a one-way random
t o t a l variance
the
item pool
receives a different
intraclass correlation ratio
large
a
the
11 C r o n b a c h e t a l . (1972, p. ratio
of
universe
score
variance. This value correlation score of
The
observed
2
to expected
can
O
( T ) /
2
and
be
and
as
[o (r) +
the
score
squared
universe
is a
coefficient
more g e n e r a l l y
as
2
variance
previously, error variance
the
a (6)]
2
score
as
t o the
such
expressed
CG
observed
equal
score
82),
CG
i s universe
CT (T)
defined
=
2
d e f i n e d the
i s approximately
( C r o n b a c h e t a l . , p.
P where
variance
between t h e
determination.
17)
and
a (6)
i s , as
2
a s s o c i a t e d with
relative
decisions. The
CG
i s comparable
reliability is
coefficients
i n t e r p r e t e d the
coefficient, measurement
p , 2
to c l a s s i c a l
same way.
n
the
score
( C r o n b a c h e t a l . , 1972, The
i n d i c a t e s how
(e.g.,
test
m a g n i t u d e of
reliably
p e r s o n s ) can
the be
p.
84)
and
the
object
rank
of
ordered
with
P
the
f a c e t s and
design. the
For
average
numbers of
example,
obtained
indicates
the
value
that
rating
i n the
of
p
=
2
f a c e t c o n d i t i o n s used
Schroeder .86
i n m a t e s can
be
assigned
two
by
e t a l . (1983)
( f o r a sample of reliably
ranked
r a t e r s on
the
inmates)
using 22
the
item
Following
guidelines
for r e l i a b i l i t y ,
acceptable
f o r r e s e a r c h p u r p o s e s where g r o u p a v e r a g e s
will
l o c u s of be
interest.
made a b o u t
(generalizability)
(1978, pp.
71
a coefficient
the
study,
checklist.
the
Nunnally's
in
245-246) of a t
least
.7
is
are
In a p p l i e d s e t t i n g s , where d e c i s i o n s
individuals, should
be
as
reliability high
as p o s s i b l e .
12
Estimating In
t h e CG
practice,
t h e CG i s e s t i m a t e d
components c a l c u l a t e d gives
i n the G study.
t h e r e s e a r c h e r an e s t i m a t e
a proposed
component
relative
a d e q u a c y of a v a r i e t y
collection
prior
accomplished in
estimates
coefficients researcher of
a
that w i l l
if
of designs
conducting
constitute
i n determining facet
the study.
of f a c e t
c o n d i t i o n s used
a ( 6 ) guides the 2
2
v a r i a n c e . In t h i s
need t o be s a m p l e d . estimate
accounts
diminished
by p l a n n i n g a D s t u d y
conditions
o r s e e k i n g ways t o d i m i n i s h t h e v a l u e
the
case
only
f o r a large
of t h e e r r o r
For
i t
In c o n t r a s t ,
proportion
variance
the
within a (6) i ssmall,
to error
component
This i s
be s a m p l e d . I f t h e e s t i m a t e of
contained
of the facet
a particular
f o r data
whether more o r fewer c o n d i t i o n s
should
contribute l i t t l e
few l e v e l s
From t h e
o f t h e CG. The m a g n i t u d e o f
a v a r i a n c e component will
procedure.
by v a r y i n g t h e number
a particular
The c a l c u l a t e d CG
t h e r e s e a r c h e r can e v a l u a t e the
to actually
t h e computation
the variance
o f t h e g e n e r a l i z a b i l i t y of
D-study data c o l l e c t i o n
variance
from
v a r i a n c e , i t s impact'can
be
w i t h a l a r g e number of of the
itself.
example,
general
p
2
form
i n the Schroeder of the e s t i m a t e d
= a /[o 2
p
2
+ d ./n.
+a
2
p
pi
2
I
e t a l . (1983) example,
CG i s e x p r e s s e d a s :
/n
pr
+ a . 2
r
pir, e
/n.n ] I
r
13 If
i t i s found
that
t h e component
a
, c o r r e s p o n d i n g t o the
2
pr person
by
rater
component raters
interaction,
can be m i n i m i z e d
i n the D s t u d y .
then
s t e p s s h o u l d be
data
by
improving
decreasing
taken
In
other t e s t i n g t i m e may
facet it the
would
s h o u l d be
here
that
noted
the
interest.
included
within
be
or because
some o t h e r
of
the
thereby
t h e c h o i c e of
facets
s t u d y . The
Schroeder
t h e e q u i v a l e n c e of stability
facets
can
suggested
raters.
of measurement
In s u c h a c a s e an
several
some f a c e t s can
so d e s i g n e d
practical,
i n the g e n e r a l i z a b i l i t y
that
this
number of
the q u a l i t y
of the
f o c u s e d on
same d e s i g n . F u r t h e r , as
example, was
be
of
i s not
of
r
situations
be
a large
procedures,
depends upon t h e p u r p o s e
a l . (1983) example
impact
.
2
noted
et
over
approach
t o improve
a
the v a l u e of
the
employing
the t r a i n i n g
s h o u l d be
studied
by
If t h i s
p
It
i s large,
be
i n an
nested, e i t h e r
Occasions study. Also
included
in
earlier
because
the
facets are n a t u r a l l y
nested
f a c e t - - s u c h as C l a s s e s n e s t e d
within
study
Schools. Generalizability The
use
Theory
Research
of g e n e r a l i z a b i l i t y
by a number of a u t h o r s w i t h i n Jackson
& Paunonen,
These a u t h o r s approach classical
1980;
stressed
to r e l i a b i l i t y test
score
the
and
Application
t h e o r y has field
Mitchell,
approach.
Wiggins,
of a
e s t i m a t i o n over
advocated
of p s y c h o l o g y
1979;
the advantages
been
the
(e.g.,
1973).
multifaceted traditional
1 4 Pedagogical the
generalizability
science 1981)
literature.
and
Rentz
the a p p l i c a t i o n Brennan and concepts an
articles
a p p r o a c h have a p p e a r e d C a r d i n e t , Tourneur,
psychological
(1979) p r e s e n t e d
approach to estimate
a l s o are
Naccarato,
1984)
In
(e.g.,
& O'Dell a
Chalmers &
Wallander,
generalizability
in multifaceted theory
instrument
applications
( e . g . , G i l l m o r e , Kane, & 1977;
(e.g., Cain
1984;
1985;
the
Nussbaum,
& Green,
198,4) and
1983;
F r a s e r , Cronshaw, &
Alexander,
literature.
properties
of
been d i r e c t e d
p.
behaviour
& Barrett,
Although
the
have u s e d
Kane & B r e n n a n ,
with
in research
M a r i o t t o , Conger, C u r r a n ,
i n the e d u c a t i o n a l
1978;
organizational Doverspike
theory
studies. Generalizability
found
essential
along
science l i t e r a t u r e .
reliability
of
application.
Hansen, T i s d e l l e , 1985)
the
theory
a number of a u t h o r s
Farrell,
1979;
design
social
literature
Conger, & Conger,
development
(1976,
demonstrations
a summary of
of g e n e r a l i z a b i l i t y
i n the
1985;
Allal
t o e d u c a t i o n a l measurement.
technique
f e a t u r e s of g e n e r a l i z a b i l i t y
s e t t i n g s appear
social
of t h e
Applications
Wallander,
and
i n the
of
detailed
example of a t w o - f a c e t
& Knight,
the a p p l i c a t i o n
(1980) p r e s e n t e d
Kane
and
demonstrating
little the
r e s e a r c h has
generalizability
toward examining
v a r i a n c e component 138)
referred
the
f o c u s e d on technique, sampling
estimates. Shavelson
to these
the
components as
the
statistical
some work
has
p r o p e r t i e s of and
Webb
"Achilles
(1981, heel"
15 of
g e n e r a l i z a b i l i t y theory
properties. sampling design
Smith
(1978) e m p i r i c a l l y
e r r o r s of
variance
models. His b a s e d on
unstable.
G studies
not
performed
linear
little
investigating neglect
relatively the
suggested provide would The then
that
large
used
sampling
of
the
Cronbach et
the
large
the
degree
were based
calculated
with
on
the
CG.
toward One
al.(l972)
coefficient;
variance
be
estimates.
they
stressed They
conducted
Such
facet
studies
from s u c h s t u d i e s
t h e s e components
are
likely
stable,
to
conditions.
g e n e r a l i z a b i l i t y of
likely
for
placed
components.
should
numbers of
reason
would
the
B e c a u s e component e s t i m a t e s b a s e d on
facet conditions
stable.
(completely
that
been d i r e c t e d
error
numbers of
the
(1966)
indicated a high
estimates
components c a l c u l a t e d
proposed design.
are
conditions
Nelson
hierarchical
component
to estimate
component
from w h i c h t o p r e d i c t
scale G studies
stable variance
variance
has
e m p h a s i s on of
two
squares.
i s that
importance
involve
be
of mean
sampling
likely
little
greater
i n the
research
the
numbers of
results also
variability
combinations
Very
this
Their
using
under
variance
s t u d i e s . Leone and
a s i m i l a r study
sampling
estimates
the
facet conditions
involving small good e s t i m a t e s
sampling
investigated
component
numbers of
subsequent D
nested) designs. of
small
provide
a d e q u a c y of
their
r e s u l t s i n d i c a t e d that
estimates
t h u s do
b e c a u s e of
the
would a l s o
CGs be
large
16 However, i n many t e s t i n g r e s o u r c e s a r e not a v a i l a b l e s t u d y . Nor, g e n e r a l l y , results available.
situations
t o conduct
are suitable
Even
i f G study
sufficient
a preliminary G
p u b l i s h e d G study r e s u l t s were
r e s e a r c h e r s would be a d v i s e d t o e x e r c i s e other
r e s e a r c h e r s ' v a r i a n c e component
e s t i m a t e s may be u n s t a b l e or
unmeasured
occasion
facets,
of t e s t i n g ,
( c f . Smith,
available,
caution i n using
e s t i m a t e s . Such 1978).
such as geographic
F u r t h e r , hidden
l o c a t i o n or
c o u l d have an impact
on t h e D s t u d y .
Most p u b l i s h e d r e s e a r c h u s i n g g e n e r a l i z a b i l i t y is
based
on a s i n g l e
data c o l l e c t i o n
D and G s t u d y . R e s e a r c h e r s t h e CG t h a n
in
reflects
of r e p o r t i n g
instrument
summarizing studies
estimates.
studies.
and v a l i d i t y
coefficients
In a d d i t i o n ,
coefficients
instrument
often
a r e needed i n
has been d e v e l o p e d f o r
purposes.
Researchers
the
reliability
t h e a d e q u a c y o f measurement
experimental
report
t e n d t o p l a c e g r e a t e r emphasis on
the long-standing psychometric
development
i n w h i c h a new
Alexander,
s e r v e s as both the
t h e y do on t h e v a r i a n c e component
This probably tradition
that
theory
(e.g., Doverspike,
Carlisi,
1983; Kane, G i l l m o r e , & C r o o k s ,
Barrett, &
1976) o f t e n
t h e e s t i m a t e d CG f o r t h e number o f c o n d i t i o n s used i n
s t u d y a s w e l l a s e s t i m a t e s f o r o t h e r numbers o f
conditions. estimated
Some a u t h o r s have t e n d e d
to treat
CGs a s i f t h e y were p a r a m e t e r
estimates. Doverspike
these
values, rather
e t a l . , f o r example,
stated
that
than
17 "...reliability raters
was
dropped
reduced
only
from
to
was
estimates
c o n s i d e r e d . To
important
to report a confidence
d o e s not
due
coefficient
study.
interval, r a n g e of
But,
by
r e s e a r c h e r s would be the
inferential studies.
would be
population
intraclass
single-sample independent
developed
confidence
crossed design
Fleiss
developed
similarly
90%
confidence
the
CG
measurement
with
the
of
likely
and
toward e x a m i n i n g for single
d e a l t with
the
facet
generalizability
and
coefficient
e q u i v a l e n t t o CGs. techniques
intervals
and
Feldt
f o r making
in generalizability
sample Shrout
and
Whalen
significance (1978; S h r o u t
approximate confidence
(1965,
for constructing
for coefficient
by H a k s t i a n
a k independent
coefficients.
estimated
i n v e s t i g a t e d t h e p r o p e r t i e s of
inferential
extended
a measure of
replication
computed
sample c o m p a r i s o n s
T h i s work was
standard
upon
provided
coefficients
of w h i c h a r e
presented
the
of an
that a
been d i r e c t e d
have
correlation
alpha—both
one-facet
they
or
parameter.
p r o p e r t i e s of CGs
se;
or
no
e r r o r i n the
indicating
found
T h e s e s t u d i e s have not per
However,
of
conclusions, i t is
A high value
reliable
number
sampling
interval
r e p o r t i n g say
Some r e s e a r c h has
1979)
such
thereby
to sampling.
i s adequately
favourable
theory,
of
n e c e s s a r i l y guarantee t h a t the
procedure
1969)
clarify
when t h e
(p. 481).
presence
f o r each c o e f f i c i e n t ,
uncertainty
the D
to the
1"
attention
error
given
10
slightly
two
alpha
(a
terminology). (1976) test &
intervals
who
for alpha
Fleiss, for six
intraclass and
Lind
correlation
coefficients.
(1982) d e r i v e d a p p r o x i m a t e v a r i a n c e and c o v a r i a n c e
expressions
for coefficient
expressions
to develop
dependent
sample a l p h a
suggested
calculated not is
inferential involving
the
more t h a n
Thus,
t h e major
focus
of i n f e r e n t i a l
The t w o - f a c e t
one f a c e t
of a
single
of the p r e s e n t
with
have
two o r more
procedures
design,
f o r CGs
developments, i t
on t h e p r e c i s i o n
o r t o make c o m p a r i s o n s among
development
estimate.
procedures
In t h e a b s e n c e o f s u c h
n o t p o s s i b l e t o comment
coefficients.
for multiple
Study
from d e s i g n s
coefficient
procedures
used the
coefficients.
above,
been d e v e l o p e d .
a l p h a . These a u t h o r s
inferential
Purpose of the Present As
More r e c e n t l y , H a k s t i a n
both
study
was
f o r t h e CG f a c e t s random,
•r
will
be u s e d
to i l l u s t r a t e ,
i n the next
developments, g r e a t e s t d e t a i l . Design
t o as
V I I ( s e e C r o n b a c h e t a l . , 1972, Ch. 2 ) , has a
broad
t o p s y c h o l o g i c a l and e d u c a t i o n a l
measurement p r o b l e m s ; involving
i t i s appropriate
multiple raters,
periods. Further, this allowing
procedure
for studies
or o b s e r v a t i o n
is relatively
of the development
without
the g e n e r a l i z a b i l i t y
crossed
observers,
design
the e x p l i c a t i o n
inferential
This design,
these
referred
range o f a p p l i c a t i o n
of
chapter,
extensive
of the
notation.
approach to designs
facets i s straightforward.
uncomplicated,
with
Extension more
1 9 The study are,
random f a c e t model was
because the author as a s s e r t e d by
notion
b e l i e v e s t h a t most
Shavelson
and
of e x c h a n g e a b i l i t y , sampled
In t h e i r other
c h o s e n as t h e
view,
i f facet
potential
treated
as
The employed
r e s e a r c h used
by H a k s t i a n
and
experimental
normalizing
Lind
the
design
t r a n s f o r m a t i o n was
sample c o e f f i c i e n t .
derive
be
facet,
a method
universe.
exchanged the
facet
similar
(1982) t o d e v e l o p
approximate variance expression
the
a larger
the
with should
be
random.
present
different
from
this
facets studied
Webb (1981) w i t h
c o n d i t i o n s can
c o n d i t i o n s of t h e
f o c u s of
for estimated
and
sampling
i n the
following chapter.
CGs
of
under a
expression for
method was
variance expressions. Details
are presented
an
models. F i r s t ,
a p p l i e d to the
Then t h e d e l t a
to that
these
used
to
procedures
Chapter Mathematical The and
development
their
Development
of the asymptotic
f o r the f u l l y - c r o s s e d
a l . ' s (1972, p . 38) D e s i g n begins
distribution
o f t h e CG and t h e n
application
coefficient
selected The
method
development two-facet
Schroeder
designs
with the
f o r other
o f t h e CG
e t a l . (1983) s t u d y
discussed previously
V I I G Study. In t h i s
case
balanced
ANOVA d e s i g n . The d a t a
analysed
as a three-way
replications
(i.e.,
f o r such
factorial
there
a
p
design
This
subjects
a design are without
i s o n l y one o b s e r v a t i o n p e r c e l l ) .
model u n d e r l y i n g t h e s e
+
each of
by e a c h o f two r a t e r s .
c a n be c h a r a c t e r i z e d a s a b e t w e e n - w i t h i n
= M
this
f o l l o w s t h e same s t e p s .
i n m a t e s was r a t e d on 22 i t e m s
X. pir
proceeds
of v a r i a n c e e x p r e s s i o n s
71
linear
random.
to obtain the variance expression.
an example of a D e s i g n
The
facets
i s a p p l i e d t o the transformed
is
layout
design--Cronbach
with a d i s c u s s i o n of the approximate
Approximate D i s t r i b u t i o n The
both
w i l l be
of a n o r m a l i z i n g t r a n s f o r m a t i o n . F o l l o w i n g
the d e l t a
The
expressions
intervals
two-facet
VII—with
This chapter
step,
variance
use i n c o n s t r u c t i n g c o n f i d e n c e
illustrated et
2
data
i s e x p r e s s e d as
+ b . + c + a b . + a c + be. + abc . , I r pi pr ir pir,e
21
where u i s t h e g r a n d
mean, a
i s the e f f e c t
due t o p e r s o n
p,
P b. 1
i s the e f f e c t
rater
r . The r e m a i n i n g
effects in
due t o i t e m
f o r the three-way
random e r r o r
formulas
ANOVA model a r e p r e s e n t e d variance
component
Throughout
these
interaction
i s confounded with
f o r the expected
this
mean s q u a r e s in Table
estimates
due t o
i n t h e model r e p r e s e n t t h e
due t o i n t e r a c t i o n s among t h e f a c t o r s .
the subscript
that
terms
i , and c i s the e f f e c t r
The e p s i l o n indicates
i n t e r a c t i o n . The
f o r the f u l l y
random
1 . The mean s q u a r e s and
are presented
i n Table 2 .
d e r i v a t i o n s i t i s c o n s i d e r e d that the
a s s u m p t i o n s o f t h e ANOVA model a r e t e n a b l e : i n d e p e n d e n c e of observations,
homogeneity
of v a r i a n c e , and u n d e r l y i n g normal
distribution. The (3) The
p
p o p u l a t i o n CG f o r t h i s
= a /[a 1 p 2
2
subscript
random model estimate
2
design i s
+ a . / n . +a -/n + a . /n.n ] . p pa l pr r pir, e l r 2
1 i s used from
for this
2
2
to distinguish
t h e CG f o r t h e f u l l y
t h e CG f o r o t h e r m o d e l s . The sample quantity, expressed
i n terms of
observed
mean s q u a r e s , i s (4)
p
First
2
1
= 1 - [(MS . + MS - MS . )/MS ] . pi pr pir,e p
c o n s i d e r the numerator of the q u a n t i t y MS
This
expression
(1 - P ) /
. + MS - MS . pi pr pir,e
i s a linear
combination
of
independent
2
Table 1 Expected
Mean S q u a r e s f o r D e s i g n E(MSp)
=
a
E(MSi)
=
a
E(MSr)
=
a
E(MSpi)
=
E(MSpr)
=
a
E(MSir)
=
a
=
a
E(MS
p i r f €
)
a
V I I w i t h b o t h F a c e t s Random
pir, e
+
n
i r ,
e
+
n
pir,
e
+
n
i
a
+
n
r
a
pir, e
+
n
i
a
pir, e
+
n
pair
P
F>ir,
€
2 i
r
>
e
rap>i
+
r ° p i P
r
P
i
P
r
+
n
i
r
+
n
i
n
pair
+
n
pnrpi
n
pair
+
n
pni^r
a
P
n
r
p
p
23 Table 2 A n a l y s i s o f V a r i a n c e R e s u l t s and E s t i m a t e d Components f o r P s y c h o p a t h y D a t a Source
Sum
of Squares
df
Mean
Variance
Square
o
2
Persons
363.4225
70
5.1918
.001 5
I terns
115.4882
21
5.4994
.031 1
Raters
.7686
1
.7686
.0001
P X I
1021 .9437
1 470
.6952
.2604
P X R
14.2996
70
.2043
.0014
I X R
11.9286
21
.5680
.0055
.1745
.1745
P X I X R
256.5032
1 470
variance
e s t i m a t e s . The e x a c t
such a combination utility
a linear
of the d i s t r i b u t i o n
of
i s t o o c o m p l i c a t e d t o be of p r a c t i c a l
(see F l e i s s ,
1946) d e v e l o p e d
form
1971).
However,
an a p p r o x i m a t i o n
combination.
Satterthwaite
(1941,
t o the d i s t r i b u t i o n
If g i s a linear
combination
of
such
of
variance estimates, g = a s * 11
+ a s + ... + a s , 2 2 k k
2
2
where t h e sample v a r i a n c e s
2
2
has e x p e c t e d
value
degrees
of freedom, and w e i g h t i n g
quantity with
rg/E(g)
r degrees
a , 2
r.
i
I
factor
i s approximately
i
a^, then the
chi-square
distributed
of f r e e d o m , (a
a
2
11
+ a o + ... + a, a ) 2 2 k k 2
2
2
r =
(Satterthwaite, Considering
1941). the denominator
o f (1 - P ) / n o t e 2
that the
quantity (n is
- 1)MS /E(MS ), P P
P
chi-square distributed
with p ~ n
Hence, t h e q u a n t i t y (1 - p ) 2
with degrees
of f r e e d o m
1
degrees
of f r e e d o m .
i s approximately
c, and v
2
g i v e n by
F
distributed
(MS V
,
. + MS pi pr
- MS
. ) pir,e
2
=
MS
2
.
+
P (n -D(n.-l) P i
MS
and
v
2
=
MS
+
2
2
P_r
1
(n
(n
P
-D(n
(n
-1)
r
P
. pir, e
-1)(n.-1)(n i • .
-1 ) r
1) . P
The
Normalization The
noted
distributional
above c o u l d
confidence u s e d by
of a F u n c t i o n
be
interval
Feldt
used
not
only
i t can
techniques
for hypothesis
a l s o be
application,
having
distribution
permits
Paulson's
applied
to a
used
the
with
b a s e d on
normal
the
developed
an
a very
1/3
latter underlying
theory.
n) a
normal
of
For
Hilferty's
for x
2
close normalization
-
variance
inferential
random v a r i a b l e s ,
(cF
of a
this was
CG.
transformation
~
)
technique
transformation
and
2
confidence
In t h i s
(1942) e x t e n d e d W i l s o n
Paulson
of
to develop
testing.
p
approximate
a p p l i c a t i o n of a v a r i e t y
f u n c t i o n of
a normalizing
an
(1 -
quantity
development
(1942) n o r m a l i z i n g
work on
the
setting
a coefficient
procedures
reason,
the
CG
manner t o t h e
( 1 9 6 5 ) . However, t h e
intervals;
F-distributed
to develop
in a s i m i l a r
permits
Paulson
the
p r o p e r t i e s of
expression
statistical
of
N(0,1),
(1931)
variables. for
where
u = a
1 =
2
c = and
the q u a n t i t i e s
associated Paulson's
with
Their
[2/(9^ )]F
1 -
2/(9v )
and
v
2
are
2
the F r a t i o .
the degrees
Hakstian
and
they
empirical results result
set the
indicated
yielded
a test
of
freedom
Whalen
development
f o r m u l t i p l e independent
the e x p r e s s i o n
Paulson's
2/(91;,),
+
2 / 3
2
transformation in their
significance simplify
2/(9»,),
alpha c term
(1976)
of a t e s t
coefficients equal
to
that a p p l i c a t i o n
with
used of (to
one).
of
good Type I e r r o r •
control. The cube
value
of
the
c term, w h i c h c o r r e c t s f o r b i a s
root transformed
present
F variate,
investigation;
a n a l y s e s . With the
i t was
was
c l o s e t o u n i t y i n the
set equal
smallest value
of
i n the
t o one
v
(n
2
for a l l
= 25)
considered
P
in the
the p r e s e n t larger
study,
the a c t u a l
samples c o n s i d e r e d ,
value
of
the v a l u e
c i s .9907. W i t h of
c
i s even
to u n i t y . F u r t h e r , p r e l i m i n a r y a n a l y s i s
of
the
scaling
transformed
CG
w i t h and
indicated
t h a t the
impact
the c o n t r o l
The
on
inclusion
D e r i v a t i o n of an
without of t h e
of Type I Asymptotic
the
factor
closer
the a d e q u a c y
had
of
factor virtually
no
error. Variance
Expression
f o r the
CG In t h e p r e s e n t p.
387)
was
used
r e s e a r c h , the d e l t a
to develop
method
the asymptotic
(Rao,
variance
1973,
expression deriving that
f o r t h e CG. The d e l t a
an a s y m p t o t i c
i s expressed
covariance
We
as a f u n c t i o n
begin
w i t h a v e c t o r , t,
statistics
i n the v e c t o r
asymptotic
1 / / 2
for a
of s t a t i s t i c s
statistic
with
a known
structure.
estimates
(n)
i s a technique f o r
variance expression
consistent
By
method
U
of the elements of the v e c t o r
theory,
- 0)
w h i c h c o n t a i n s u n b i a s e d and
t
6.
The
have known c o v a r i a n c e m a t r i x I .
the vector
~ MVN(0,
I ) , where n i s t h e sample
size.
1 /3 The
statistic
delta
f ( t ) = (1 - p )
tie).
estimates
2
By t h e
method, [f (t)
where
a
- f(6) ] ~ N ( 0 ,
= 'L4>. The v e c t o r
2
derivatives,
9f(t)/9t|
i or
r
005
025
050
950
975
995
a: 10
05
01
2 2 2 2 2 2 2 2 2 5 5 5 5 5 5 5 5 5 8 8 8 8 8 8 8 8 8
013 014 012 012 009 010 009 004 007 004 007 006 004 005 008 004 005 005 004 004 004 003 004 004 002 003 005
038 038 045 036 032 039 029 029 027 022 025 025 027 024 027 024 023 022 019 028 022 024 020 021 017 021 021
060 062 068 064 060 066 055 052 052 045 044 042 049 044 053 053 049 044 036 046 040 048 041 044 042 048 050
960 966 965 953 954 959 955 956 958 944 940 944 942 942 940 948 945 946 941 948 946 945 946 950 954 949 947
981 985 990 982 980 984 978 978 981 974 971 976 972 972 975 975 973 972 972 972 972 974 975 978 976 975 978
997 1000 999 1000 998 998 998 997 998 997 993 995 993 996 996 996 994 995 993 995 997 996 996 996 997 996 997
100 096 102 110 106 106 100 095 094 101 104 098 107 102 112 106 104 099 095 098 093 103 095 094 089 099 103
057 053 056 055 052 055 051 051 046 049 054 048 055 053 052 049 050 050 047 056 050 050 046 043 041 046 043
016 014 013 012 011 012 011 007 009 007 014 011 011 009 012 008 011 010 011 009 007 007 008 008 005 007 008
%
006
027
050
950
977
996
100
050
010
91.7
75.8
71.0
91.6
23.0
25.2
45.1
n
Overall Chi -square (df = 27) 144.1 131.1
Note. Decimal points omitted i n alphas and proportions. The standardized estimate i s the value [
(
1
_ £
2
)
1 / 3
.
( i -
p
2 ) 1 / 3
]
/
[
^
(
l
.
3
2
)
1 / 3
]
1 / 2 .
Table 5 Proportion of the Standardized Estimates Falling below Selected Percentile Points of the Unit Normal Distribution and Actual Proportion of Type I Errors for p = .70 (Design VII, both Facets Random) z
"p
25 25 25 75 75 75 150 150 150 25 25 25 75 75 75 150 150 150 25 25 25 75 75 75 150 150 150
i
n
10 20 30 10 20 30 10 20 30 10 20 30 10 20 30 10 20 30 10 20 30 10 20 30 10 20 30
n
Mean p i
o
r
p: 005
025
050
950
975
995
a: 10
05
01
2 2 2 2 2 2 2 2 2 5 5 5 5 5 5 5 5 5 8 8 8 8 8 8 8 8 8
013 014 018 006 006 014 005 007 008 004 004 005 004 004 007 003 006 005 005 005 006 006 004 003 004 004 005
045 043 046 038 024 042 031 031 030 022 020 022 024 025 026 021 030 025 023 022 024 023 022 021 022 025 021
070 073 070 065 055 070 062 053 054 040 044 045 054 048 052 047 056 044 048 046 044 043 049 047 044 050 043
964 958 956 963 962 958 959 964 950 946 951 937 944 951 953 952 955 945 933 939 949 951 944 943 941 944 947
986 982 979 984 983 982 984 980 974 973 977 971 970 974 978 974 979 973 970 972 974 974 968 976 968 974 976
998 996 999 998 998 997 996 996 995 994 997 994 994 998 998 994 "994 994 993 996 994 994 994 997 994 996 997
106 114 114 102 093 112 103 090 104 094 092 108 110 097 098 095 100 100 115 107 094 092 105 104 103 107 096
058 060 067 054 041 060 048 050 056 049 043 051 054 052 048 048 051 052 053 050 050 049 053 045 054 051 045
015 018 019 008 008 017 009 011 013 010 007 011 010 006 009 009 012 011 012 009 012 012 010 006 010 008 008
i
006
028
052 . 950
976
996
102
052
011
76.5
60.2
42.1
44.5
66.8
a
f
Overall Chi -square (df = 27) 174.5
169.4
132.4
Note. Decimal points omitted in The standardized estimate is the [(1-p ) 2
1 / 3
-
(1-p ) 2
1 / 3
97.8
alphas and proportions. value
]/[Var(1-p ) / ] / . 2
1
3
1
2
Table 6 Proportion of the Standardized Estimates Falling below Selected Percentile Points of the Unit Normal Distribution and Actual Proportion of Type I Errors for p * = .90 (Design VII , both Facets Random) L
n
25 10 25 20 25 30 75 10 75 20 75 30 150 10 150 20 150 30 25 10 25 20 25 30 75 10 75 20 75 30 150 10 150 20 150 30 25 10 25 20 25 30 75 . 10 75 20 75 30 150 10 150 20 150 30
n
n
p
Mean p^ or
p: 005
025
050
950
975
995
a: 10
05
01
2 2 2 2 2 2 2 2 2 5 5 5 5 5 5 5 5 5 8 8 8 8 8 8 8 8 8
016 014 018 010 009 006 009 006 011 004 006 008 004 004 006 006 007 008 003 004 006 003 002 005 004 008 007
047 043 054 033 034 026 031 028 035 022 027 028 022 022 030 022 021 026 024 022 021 022 023 025 020 024 026
078 074 084 060 060 047 060 054 060 041 048 046 043 052 058 045 048 054 043 042 040 037 047 048 046 052 040
969 962 964 959 958 956 958 952 958 945 943 951 936 943 948 947 956 946 935 940 941 945 941 940 949 951 946
987 986 986 979 980 984 980 979 981 973 974 978 970 972 972 975 976 976 966 968 970 976 969 975 974 980 972
1000 999 1000 998 998 996 996 997 997 994 995 995 996 994 996 994 997 997 992 994 994 995 993 995 996 997 996
109 113 120 101 102 090 102 102 102 096 105 095 106 108 109 098 092 108 108 102 100 092 106 108 097 100 094
060 058 068 054 054 042 051 049 054 049 053 050 053 050 057 047 045 050 058 054 050 046 054 050 046 045 054
016 015 018 012 011 010 013 009 013 010 011 013 008 010 010 012 101 011 011 010 012 008 009 010 008 011 011
i
007
028
052
950
976
996
102
052
011
192.0 188.4
106.3
95.4
80.3
38.4
43.2
40.9
a
r
Overall Chi -square (df = 27) 189.2
Note. Decimal points omitted in
alphas and proportions.
The standardized estimate is the [
(
l
-
p
2 ) l / 3
_ ( i - 2 ) 1 / 3 ] p
/
[
^
value N
a
r
(
l
- 2 1 / 3 1 / 2 ?
)
]
