This paper has been published. Please cite it as: González, J., Barrientos, A. F., & Quintana, F. A. (2015). A Dependent Bayesian Nonparametric Model for Test Equating. In Millsap, R.E., Bolt, D.M., van der Ark, L.A., Wang, W.-C. (Eds.) Quantitative Psychology Research, pp 213-226. Springer.
A Dependent Bayesian Nonparametric Model for Test Equating Jorge Gonz´alez, Andr´es F. Barrientos and Fernando A. Quintana
Abstract Equating methods utilize functions to transform scores on two or more versions of a test, so that they can be compared and used interchangeably. In common practice, traditional methods of equating use parametric models where, apart from the test scores themselves, no additional information is used for the estimation of the equating transformation. We propose a flexible Bayesian nonparametric model for test equating which allows the use of covariates in the estimation of the score distribution functions that lead to the equating transformation. A major feature of this approach is that the complete shape of the score distribution may change as a function of the covariates. As a consequence, the form of the equating transformation can change according to covariate values. We discuss applications of the proposed model to real and simulated data. We conclude that our method has good performance compared to alternative approaches.
1 Introduction Equating is a family of statistical models and methods that are used to make test scores comparable on two or more versions of a test, so that scores on these different test forms, intended to measure the same attribute, may be used interchangeably (see, e.g. Holland and Rubin, 1982; Kolen and Brennan, 2004; von Davier et al., Jorge Gonz´alez Faculty of Mathematics / Measurement Center MIDE UC, Pontificia Universidad Cat´olica de Chile, Av. Vicu˜na Mackenna 4860, Macul, Santiago, Chile, e-mail:
[email protected] Andr´es F. Barrientos Faculty of Mathematics, Pontificia Universidad Cat´olica de Chile, Av. Vicu˜na Mackenna 4860, Macul, Santiago, Chile, e-mail:
[email protected] Fernando A. Quintana Faculty of Mathematics, Pontificia Universidad Cat´olica de Chile, Av. Vicu˜na Mackenna 4860, Macul, Santiago, Chile, e-mail:
[email protected]
1
2
Jorge Gonz´alez, Andr´es F. Barrientos and Fernando A. Quintana
2004; Dorans et al., 2007; von Davier, 2011). To avoid the confounding of differences in form difficulty with those of test takers’ abilities, different equating designs to collect data are used. Once these effects are corrected, the purpose of equating is to obtain comparable scores for both groups, by adjusting for differences in difficulty of the test forms. Let Tx and Ty be the random variables denoting the scores on tests X and Y which are to be equated, with associated cumulative distributions functions (c.d.f) Sx = S(tx ) and Sy = S(ty ), respectively. In what follows we assume that scores on X are equated to the scale of scores on Y, but arguments and formulas for the reverse equating are analogous. Let tx and ty be the quantiles in the distributions of tests X and Y for an arbitrary common cumulative proportion p of the population, such that −1 −1 tx = Sx (p) and ty = Sy (p). It follows that an equivalent score ty on test Y for a score tx on X can be obtained as −1
ty = ϕ(tx ) = Sy (Sx (tx )).
(1)
In the equating literature, function ϕ(tx ) is known as the equipercentile transformation. A graphical representation of the equipercentile method of equating is shown in Figure 1. Note that because ϕ(tx ) is built from distribution functions, the equipercentile equating method is nonparametric by nature. Because sum scores (i.e, total number of correct answers) are typically used in measurement programs, an evident problem with (1) is the discreteness of the score
0
p
Cumulative Distribution Function
1
S^y S^x
t_y
t_x
Scores
Fig. 1 Graphical representation of equipercentile equating. A score tx in test X is mapped into a −1 score on the scale of test Y using ty = ϕ(tx ) = Sy (Sx (tx ))
A Dependent Bayesian Nonparametric Model for Test Equating
3
distributions, rendering their corresponding inverses unavailable. The common solution given to this problem in the equating literature is to actually “continuize” the discrete score distributions Sx and Sy , so that (1) may be properly used for equating. In many applications, complementary information besides the test scores themselves is available most of the time (e.g., examinee gender, type of school, point in time of the administration, etc), yet the use of covariates seems to be a rather unexplored topic in the equating literature. It is natural to think that the information provided by covariates could improve the equating task. Additionally, despite the nonparametric nature of the transformation ϕ(tx ), the problem of obtaining a point estimate of it has traditionally relied on either parametric or semi-parametric models (Gonz´alez and von Davier, 2013). For instance, in the linear equating transformation (Kolen and Brennan, 2004) both Sx and Sy are assumed to be a locationσ scale family of distributions leading to ϕ(tx ; π) = µy + σxy [tx − µx ] where in this case π = (µX , µY , σX , σY ) are the means and standard deviations of the two score distributions. Constraining the inference to a specific parametric form, however, may limit the scope and type of inferences that can be drawn. Indeed, in many practical situations, a parametric model could not describe in a proper way the observed data. Bayesian nonparametric (BNP) generalization of parametric statistical models (see, e.g., Ghosh and Ramamoorthi, 2003; M¨uller and Quintana, 2004; Hjort et al., 2010) allow the user to gain model flexibility and robustness against mis-specification of a parametric statistical model. See M¨uller and Mitra (2013) who give many examples that highlight typical situations where parametric inference might run into limitations, and BNP can offer a way out. In this paper we propose a flexible Bayesian nonparametric model for test equating which allows the use of covariates in the estimation of the score distribution functions that lead to the equating transformation. A major feature of this approach, compared to other traditional methods, is that not only the location but the complete shape of the score distribution may change as a function of the covariates. The rest of this paper is organized as follows. We briefly review the Bayesian nonparametric modeling approach in Section 2, presenting the dependent BNP model for test equating which allows the use of covariates. Section 3 illustrates the uses and applications of the model in both simulated and the real data. The paper finishes in Section 4 with conclusions and discussions.
2 Bayesian nonparametric modeling In this section we present the proposed model, including a brief description of nonparametric models. We develop the material to the extent needed for clarity of presentation.
4
Jorge Gonz´alez, Andr´es F. Barrientos and Fernando A. Quintana
2.1 Nonparametric models Statistical models assume that observed data are the realization of random variables following some probability distribution. Let x1 , . . . , xn be observed data defined on a sample space X , and distributed according to a probability distribution Fθ , belonging to a known family F = {Fθ : θ ∈ Θ }. This setup is referred to as a parametric model whenever Θ is assumed to be a subset of a finite dimensional space. In the parametric Bayesian framework (e.g., Gelman et al., 2003), a prior p(θ ) is defined on Θ . Parametric Bayesian inference is then based on the posterior distribution p(θ | x), which is proportional to the product of the prior p(θ ) and the likelihood p(x | θ ). Although the parametric approach is adequate in many situations, it could not be realistic in many others. For instance, under a normal model, all we can possibly learn about the distribution is determined by its mean and variance. The nonparametric approach starts by focusing on spaces of distribution functions, so that uncertainty is expressed on F itself. Of course, the prior distribution p(F) should now be defined on the space F of all distribution functions defined on X . If X is an infinite set then F is infinite-dimensional, and the corresponding prior model p(F) on F is termed nonparametric. The prior probability model is also referred to as a random probability measure (RPM), and it essentially corresponds to a distribution on the space of all distributions on the set X . Thus Bayesian nonparametric models are probability models defined on a function space (M¨uller and Quintana, 2004). These models are dealt with in the same spirit as the usual parametric Bayesian models, and all inferences are based on the implied posterior distribution. BNP methods have been the subject of intense research over the past few decades. For a detailed account, we refer the reader to Dey et al. (1998), Ghosh and Ramamoorthi (2003), and Hjort et al. (2010).
2.2 The Dirichlet process (DP) and the DP mixture (DPM) model. The most popular RPM used in BNP is the DP introduced by Ferguson (1973). We say that F is a DP with parameters m and F ∗ , denoted as F ∼ DP(m, F ∗ ), if for every partition of the sample space A1 , . . . , A p , F(A1 ), . . . , F(A p ) is distributed as Dir(mF ∗ (A1 ), . . . , mF ∗ (A p )). Here, F ∗ is a base measure specifying the mean, E(F) = F ∗ , and m is a parameter that helps in determining the uncertainty of F. The DP is a conjugate prior which means that, given the data, the posterior distribution of F is also a DP. A convenient way to express the DP is via Sethuraman’s (1994) representation, which states that F ∼ DP(m, F ∗ ) can be constructed as ∞
F(·) = ∑ ωi δθi (·), i=1
(2)
A Dependent Bayesian Nonparametric Model for Test Equating iid
5 iid
where δθi (·) denotes a point mass at θi , θi ∼ F ∗ , ωi = Ui ∏ j
