The Fitting of Straight Lines when Both Variables are

Journal of the American Statistical Association

ISSN: 0162-1459 (Print) 1537-274X (Online) Journal homepage: http://amstat.tandfonline.com/loi/uasa20

The Fitting of Straight Lines when Both Variables are Subject to Error Albert Madansky To cite this article: Albert Madansky (1959) The Fitting of Straight Lines when Both Variables are Subject to Error, Journal of the American Statistical Association, 54:285, 173-205 To link to this article: http://dx.doi.org/10.1080/01621459.1959.10501505

Published online: 11 Apr 2012.

Submit your article to this journal

Article views: 25

Citing articles: 57 View citing articles

Full Terms & Conditions of access and use can be found at http://amstat.tandfonline.com/action/journalInformation?journalCode=uasa20 Download by: [71.74.122.179]

Date: 08 January 2016, At: 16:09

THE FITTING OF STRAIGHT LINES WHEN BOTH VARIABLES ARE SUBJECT TO ERROR* ALBERT MADANSKY

Downloaded by [71.74.122.179] at 16:09 08 January 2016

RAND Corporation Consider the situation where X and Yare related by Y = a +j3X, where a and j3 are unknown and where we observe X and Y with error, i.e., we observe z=X+u and y=Y+v. Assume that Eu=Ev=O and that the errors (u and v) are uncorrelated with the true values (X and Y). We survey and comment on the solutions to the problem of obtaining consistent estimates of a and j3 from a sample of (x, y)'s, (1) when one makes various assumptions about properties of the errors and the true values other than those mentioned above, and (2) when one has various kinds of "additional information" which aids in constructing these consistent estimates. The problems of obtaining confidence intervals for fJ and of testing hypotheses about fJ are not discussed, though approximate variances of some of the estimates of j3 are given. 1.

INTRODUCTION . . . . . . . . . . . . . . . . . . • . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . "

1.1. Regression, Structural and Functional Relationships. ... . .. .. .. 1.2. Least Squares and Maximum Likelihood Estimation... . . .. ... 2. 3.

. ..

THE METHOD OF GROUPING. • . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. USE OF INSTRUMENTAL VARIABLES... . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ..

3.1. Two Linearly Related Instrumental Variables Observed with Error. . . . .. 3.2. One Instrumental Variable Observed Without Error " 4.

USE OF VARIANCE COMPONENTS..........................................

4.1. Replication of Observations 4.2. The Method of Grouping 4.3. Use of Instrumental Variables

" " :. . . . . ..

5.

THE BERKSON MODEL. . . . • . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ..

6. 7. 8. 9.

ESTIMATION VIA CUMULANTS. . . . • . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. ESTIMATION IN IDENTIFIABLE CASES

'"

..

AN EXAMPLE.....

.. .. .

ApPENDIX. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ..

REFERENCES. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . ..

173 175 177 181 185 186 187 189 189 191 191 194 195 196 198 200 203

1. INTRODUCTION

physicist, say, were to give a statistician a set of observations on two variables, tell him that only one of the two variables is subject to "error" (where this "error" may be due to either errors in observation or random variation, or perhaps both), and ask him to "fit a straight line to the data," the statistician would only have to know how the observations were obtained, certain properties of the variables, that a linear relation exists, and the use to which the line is to be put, and, in the light of this information, he could fit the desired straight line to the data. If the same physicist were to come to the statistician with a set of observations on two variables and were to say that there were errors made in observing both of the variables, he would be surprised to hear the statistician request, in addition to the information mentioned above * This paper is an outgrowth of a Master's 'I'hesis submitted to the Department of Statistics, University of

I

F A

Chicago. I am indebted for helpful commente and criticisms to T. E. Harris, W. H. Kruskal, L. J. Savage, and especially to D. L. W..U..ce•. ~ aleo wish to thank Arthur Stein for making .." ..ilable to lne the date used in Section 8.

Downloaded by [71.74.122.179] at 16:09 08 January 2016

174

AMERICAN STATISTICAL ASSOCIATION JOURNAL, MARCH 1959

(which the physicist is presumably able to give the statistician quite readily), either more information on the range or standard deviation of the errors, observations on a third set of variables related to the other two, or replications of observations on each independent variable. In addition, he would be astounded to see the statistician quake if he were to mention that he believed the errors to be distributed normally. To the physicist's eye, the situation in which both variables are subject to error does not seem to be quite as intractable as the statistician makes it out to be. If he plots the data, he could certainly plot an "eye-line" which, although it may not be the "best" line (in some sense), may be pretty close to the best line to be found by a statistician. If he knew of the method of least squares, he might argue that there are cases where for a large sample the estimate of the slope of the line relating Y to X lies between the least squares estimate of the slope of the line relating the observed y to the observed x and the reciprocal of the least squares estimate of the slope of the line relating the observed x to the observed y,l and hence an averaging of these two quantities calculated from the observations would lead to an "estimate" of the true slope. Hence to the physicist the only job left for the statistician is to make a more precise estimate than the one the physicist has already made, that is, use an estimator with known "nice" properties (e.g., consistency), and attach a standard error to the estimate made. Assuming that the physicist's initial shock upon hearing that although he has an "estimate," the statistician doesn't, is over, and that the physicist is willing to hear what the statistician has to say for himself, what sort of things would the statistician like to know? As in the situation in which there were no errors in either variable, or where only one variable was subject to error, he would like to know the answers to some preliminary questions designed to give him an understanding of the problem. First of all, he would like to know something about the situation out of which the observations arose, e.g., were the observations random pairs? Is the underlying linear relationship symmetric in X and Y (in a sense to be defined later) or not? Secondly, he would like to know the use to which the linear relation is to be put, e.g., does the physicist merely want an estimate of the parameters of the linear relation? Or is he trying to predict something by using the relation? Or does he want to test some hypothesis about one or more of the values of the parameters of the linear equations? Finally, the statistician would like to know the characteristics of the underlying true variables, e.g., are they fixed numbers? Or are they random? If so, what can we assume about their distribution? With this preliminary information, the statistician can then determine the type of relationship in which the physicist is interested ... and then it only may be possible to obtain a consistent estimate of the linear relationship. Besides this orientation information which the physicist is prepared to give the statistician, the statistician, as we shall soon see, will probably also need technical information about the errors which the physicist is probably unprepared to give, because· he doesn't expect to be asked for such information. 1

For a proof of this sometimes useful fact, ei. [16J and [29J.

175

Downloaded by [71.74.122.179] at 16:09 08 January 2016

FITTING OF STRAIGHT LINES

1.1. Regression, Structural and Functional Relationships What are the possible situations out of which the observations arise? One situation is the following: Let X and Z be the true values which we are trying to observe. Suppose the distribution of Z given X is normal with mean a+~X and variance (/"2. Then, regardless of whether X is a random or a fixed variable, we can write, for fixed X, ZJ X =a+~X +t where t is normally distributed with mean zero and variance (/"2. (So far, this is the ordinary linear regression situation with no errors in observing either variable. Rather, Z is subject to random variation.) However, we do not observe z=Z or x=X but instead z = Z +v and x = X +u. If we let Y = a+~X, then our observation on Z for a given X (but not for a given z) can be written as y= Y +t+v. It is assumed that Eu =Ev =0 and that u, v, and tare uncorrelated with each other and with X and Y. This situation is usually called the regression situation [21, 27]. It is essential that t~O for this model. The case where t=O is considered separately. One important property of the situation when X is a random variable is its asymmetry. By this I mean that although Y =E(Z\ X) =a+~X, the expression X = (Y -a)/~, a result of algebraic manipulation of the original equation, is not a meaningful relation in this context. The only meaningful "inverse relation" here is Y'=E(XjZ). In particular, when the joint distribution of X and Z is bivariate normal, then E(X\ Z) ='Y+oZ, which is not the result of solving the equation Y =a+~X for X. Another important point to note is that when we write y = Y +t+v, we must distinguish between t and v. The variable v is an error of observation which we presumably may be rid of by making finer and finer observations, whereas even if we were rid of v, we would never be rid of t. The variable t is inextricably tied up to the distribution of Z given X. The variable v is what Tukey [45] would call "fluctuation" and t is what he would call the "individual part" of an observed quantity. Finally, one should note that it makes no difference in the case in which there is no error in X whether we fix X and observe X or choose random pairs of observations, for in either case, in the relation Y =E(ZI X) =a+~X, X is not treated as a random variable in considering the linear relationship of interest, namely the expectation of Z given X. When X is observed with error, we can still select our pairs of observations either as random pairs or by fixing the value of x and observing the corresponding y. For example, suppose we wish to estimate s, the density of iron, by making use of the relation MASS=jJ VOLUME. We can either select pieces of iron at random, so that our pairs of observations are random, or select pieces of iron of predetermined volumes, where the volumes are measured by some technique which yields the true volume of a piece of iron plus some random error. In this case, it makes a great deal of difference whether or not we can fix x in obtaining our observations, as we shall see later. Another situation in which one might be interested in a linear relation between variables X and Y is the following. Let X and Y be the true values which we are trying to observe, and let them be linearly related by Y =a+~X. In this case X, and hence Y, may be either random or nonrandom variables. We observe y = Y +v and x = X +u. Again, it is assumed that Eu = Ev = 0 and

zi

176

AMERICAN STATISTICAL ASSOCIATION JOURNAL, MARCH 1959

Downloaded by [71.74.122.179] at 16:09 08 January 2016

further that in situations where X and Yare random variables, EYv=EXu=O. If X is a random variable, the relation Y =a+,BX is usually called a structural relation. (However, Lindley [27] calls this a functional relation. It is in this situation that "confluence analysis" [17, p. 525] is applied.) Y = a+,BX has been called a functional relation by Kendall [21, 22] when X is not random. One immediately notes that this latter situation is a degenerate case of the regression situation, in that here t=O. One also notes that this situation is symmetric in that X=(Y-a)/,B=oy+oY is an equally meaningful way of writing the relation Y =a+,BX in this context. In any given situation, though, it may be difficult to determine whether to treat the relation as structural or functional. In the above example, for instance, we might interpret the linear relation as a functional relation by assuming that the pieces of iron we use are not a random sample from the population of pieces of iron, but merely what iron we had available and that, for a given piece of iron, there is a true mass and a true volume which we observe with random error due to the inaccuracies of our measuring devices. On the other hand, we might assume that the iron we had available was a random sample from the population of pieces of iron, so that the true mass and true volume of any piece of iron is a random variable. The determination of whether one treats the relation as structural or as functional depends on what sort of inferences one wishes to make. In this case, our treatment depends on whether we wish to estimate the density of iron or the density of the iron in our backyard. What are some of the reasons for wanting to estimate a and ,B? We may be interested in estimating a and ,B because we are interested in the structural relation between two variables and hence these values are of intrinsic interest to us. For example, we may be considering the relation MASS =13 VOLUME for a given element and desire to estimate 13, the density of the element. Or we may have some hypothesis about the values of a and 13 and might need estimates of these quantities for use in the statistic to be used to test this hypothesis. Or we might be interested in estimating a and 13 for predictive purposes. That is to say, we may at some future time want to observe a value of X without error and use the relation Y =a+,BX to predict Y from this X. From the symmetry of the structural and functional relations, if we want to do so, we can also observe a Y without error and predict X from the equation X = (Y -a)/,B. This, however, cannot be done in the regression situation, as has been pointed out earlier. One should note that there is another problem in this context which is also called the prediction problem. This is the situation in which one can never hope to observe X Or Y without error, and hence is only interested in predicting Ey = Y for a new observed x = X +u. But this is just the case in which the leastsquares regression of y on x works, for our independent variable is no longer X but instead x, which is observed without error. In this case, the statistician has no difficulty in estimating the parameters of the linear relation of interest. (Cf. [11, 27, 48] on this point. The confusion here is a result of and a good example of the confusion between regression and structure-function.)

177

FITTING OF STRAIGHT LINES

Downloaded by [71.74.122.179] at 16:09 08 January 2016

1.2. Least Squares and Maximum Likelihood Estimation

I have spent some time outlining the preliminary information necessary to the statistician before he can undertake to estimate a linear relation between two variables observed with or without error. But once the statistician has this preliminary information, thereby ascertaining what type of linear relationship obtains, he still cannot estimate the linear relationship when both variables are observed with error. Let us say that we observe Xi=Xi+Ui and Yi= Yi+Vi where Y i=a+.8Xi, and assume that EUi=Evi=O, that the errors (Ui and Vi) are uncorrelated with each other and with the true values (Xi, Y i), that our successive observations are independent, and that Var X i=U'X2, Var Ui=U'u 2, Var Vi=U'v 2, Var Xi=U',,2, Var Yi=U'y2, and Var Y i=U'y2 for all i. Then using ordinary least squares techniques (i.e., minimizing LWi(Yi-a-(3Xi)2 where Wi is the reciprocal of the variance of Yi-a-.8Xi given Xi, i.e., wi=1/U'.2) is not correct, for use of this method yields efficient, consistent estimates of .8/(1 + (U',NU'x 2)), not of .8. By using ordinary least squares techniques, we are only minimizing "vertical" error, error in the Y direction. Our situation is such that we also have "horizontal" error which should be taken into account in estimating .8. To use least squares estimation correctly, as Lindley [27] points out, one should take account of both errors by minimizing r,

L

Wi(f1)(Yi - a - f1Xi)2

i=l

where the Wi(.8)'S are proportional to the reciprocals of the variance of Yi-a -.8Xi given Xi, i.e., Wi({1) = k/(U'v 2+.8 2U'u2) , where k does not depend on i. If we knew "A = U'v 2/U'u2, then Var (Yi-a-(3Xi) = U'v 2+(32U'u2= ("A +(12)U'u2. Hence if Wi((3) =1/("A+.8 2 ) , we could minimize LWi({1)(Yi-a-.8XY with respect to (3 quite readily, and obtain equation (3) below as our estimate of f1. Lindley points out that this is the same estimate as that obtained by minimizing the distance between (Xi, Yi) and (Xi, Y i) for all i. The method of weighted least squares, with weights depending on .8, will also give an estimate of .8 if either U'u 2 or U'v 2, or both, rather than "A=U'.2/U'u 2, is known. If one assumes that X, Y, u, and V are each normally distributed, with Eu=Ev=Euv=EXu=EXv =EYu=EYv=O, then, as will be seen later, the method of maximum likelihood will give the same estimate as that obtained by the method of weighted least squares, whatever one of the above assumptions is made about U'u 2 and U'v 2 , but not without one of these assumptions. Thus, to use standard statistical techniques of estimation to estimate f1, one needs additional information about the variance of the errors. Let us see exactly where the difficulty arises in using the aforementioned techniques. I shall consider the structural relation in detail here. The analysis of the regression situation is given by Lindley [27], and of the functional relation by Kendall [21]. Suppose we observe a random sample, (Xl, YI), ... , (Xn, Yn). Let EXi=EX i =p., EYi=EY i=E(a+(3Xi) =a+f1p., U',,2=U'X 2+U'u2, U'y2=(32U'X 2+U'v2, and

178

AMERICAN STATISTICAL ASSOCIATION JOURNAL, MARCH 1959

COY (Xi, Yi) = (1UX 2. Let the Xi and Yi be normally distributed with these parameters. We then have six parameters, namely f.L, «s", u,,,z, u v 2, a, and (1. (Note that we already have assumed COY (U'i, Vi) =0.) Our sufficient statistic is a quintuple, namely (LX, LY, Lx 2 , Ly 2 , LXY). We are mainly interested in estimating {3=Cov (X, Y)/ux 2. But we can only estimate f.L, a+{3f.L, u,,} =UX2+U,,2, Uy2=uy2+uv2, and Cov (x, y) ={3ux2=Cov (X, Y). The maximum likelihood estimates of the parameters of the distribution of (X, Y) are {i.=x, &=y-Sx, o-x2= o-z2_0-,,2, o-y2=o-y2_o-V2, and Cev (X, Y)=Cov (x, y). But Uy2={32ux2, so, disregarding the equations in & and il, our equations become:

Downloaded by [71.74.122.179] at 16:09 08 January 2016

and COY (z, y) = So-x2.

We therefore have three equations in four unknowns, namely S, o-x2, 0-,,2, and o-v 2. Hence if we knew either uu2, u v2, or u v2/U,,2 and were sure that Cov (Ui, Vi) =0, we could estimate (1. With this estimate of (1, we can always estimate a by &=y-Sx. If u v 2 is known, we have from the above equations n

L

(Yi - y)2 - nuv 2

i=l

S=------n

(1)

L (Xi - X)(Yi - y) i=1

If

uu2

is known, n

S=

L (Xi - X)(Yi - y) _'_'~_1

_

n

L

(Xi - X)2 -

(2)

nu u 2

i=l

If "A=uv 2/U,,2 is known, {3 is estimated by

n

(3)

2 L (Xi-X)(Yi-Y) i=l

(Lindley's estimate [27] is incorrect. See Appendix and [6].) One can easily verify directly that the method of weighted least squares estimation yields the same estimate of {3. Smith ([40], p. 12) gives an elegant verification of this. If both u,,2 and u v2 are known, we need not compute "A but instead can use these bits of information separately and obtain another estimate of {3, namely

179

FITTING OF STRAIGHT LINES

"

}: is. -

y)2 - n
Reiersol [36] considers the case in which one knows the constants 1'1 and 1'2 of the relation 'YIZl+'Y2Z2=O between two sets of instrumental variables, ZI and Z2. For convenience, let us change notation for the moment. Let X =X1, Y =X2, and rewrite the structural relation Y =fJX as fJ lXl+fJ2X 2=O where fJ= -fJI/fJ2 and EX l=EX2=O.3 Also, let U=Ul, V=U2, X=Xl, and y=X2. Consider the situation in which we observe the instrumental set with error, i.e., ooj=Zj+wh j=i, 2. Our observations are the quadruples (Xli, X2i, ooli, oo2i), i = 1, . . . , n. Assume, as usual, that the errors in all our variables are uncorrelated with the true values. Define Ai = E (U,Wi) = E (Uiooi) = E (X,Wi) , ilij = E(Xiooj) , and J.l.ij = E(xiZj). Then fJ lXl+fJ2X2=O implies that fJ 1E(oo j X 1) +fJ2E(oo jX2) =0, or fJlPlj+{32P2j=O. If we define B = (fJl, fJ2) and P

=

U (il P12) il12 P22 '

then Bjl=O. But p=J.I.-A where J.I. = (J.l.U J.l.12) ,

J.l.2l -s:

J.l.22

• If we define X;' =X; and Y;', = Y; -Y, we see that X;' ~pY;', or P.X;' +/l2Y;' =0 for all i, where P= -Pl/h and EX", =EYi' =0. Since we know how to estimate a once fJ is estimated, we can, without loss of generality, con.. aider the above homogeneous linear relation.

187

FITTING OF STRAIGHT LINES

and

so that B([.L-X) =0. It is also evident that ,ui1'Y1+,ui2'Y2=0. If we define r =('1'1, '1'2), then ,ur' = ([.L-X)r' =0. But ([.L _ A)r' = ('Y1JLll 'Y2[.L12

+ 'Y2[.L12 + 'Y2[.L22 -

'Y 1A1) = (0)'" 'Y2A2 0

Downloaded by [71.74.122.179] at 16:09 08 January 2016

implies that Xi = ([.Lr');/'1' i, where ([.Lr') i is the i-th element of the vector [.Lr'. One sees that

and hence B([.L-X)=O implies that (32[.L21=((31[.L12"Y2)/'Y1 and so (3=-(31/(32 = - ('Y1[.L21) / ('Y2[.L12). Since we know '1'1 and '1'2, and since m12 = (

E

X1iZ2i) /

nand m21 = (

E

X2iZ1i) /

n

are consistent estimates of [.L12 and [.L21, respectively, (3 is estimated by - (''/lm21)/(''Ydm12) , provided that Cov (z, x) ~O. 3.2. One Instrumental Variable Observed without Error The great difficulty with the above method is that we not only need a pair of instrumental variables, but also a knowledge of the linear relation between them. Let us consider the case in which we have observations on only one instrumental variable, say Z, and where, in contrast to the Reiersol situation, Z is observed without error. Consider once again the relation (31X i+(32Yi=0, where (3= -(31/(32, multiply it by Zi/n and sum over all i, to obtain the expression

Call the left hand side (31"'1X+(32"'1Y. Consider the same expression with y and x substituted for Y and X, and call the left hand side (31"'1",+(32"'1Y' Then "'11/ -

"'IY = (

E

ZiVi) /

n and "'I", - "'Ix = (

E

ZiUi) /

n.

E("'I"'-"'Ix)=E("'IY-"'IY) = 0 and Var ("'I"'-"'Ix)=O(l/n), Var ("'IY-"'Iy)=O(l/n). Hence 7}y-"'IY and "'I",-"'Ix converge to zero in probability so that (31 "'I",+(32"'1Y =0

is a consistent estimate of the true relation, and n

LZiYi b=

_'_'~_1_ _ n

LZiYi

188

AMERICAN STATISTICAL ASSOCIATION JOURNAL, MARCH 1959

is an estimate of f3 = -

f3I! f32, provided that n

L

Z;x;

;=1

doesn't approach zero as n~ 00, i.e., Cov (Z, x) :;=0. Besides obtaining this estimate of f3, Geary [14] also derives the exact distribution of a function of b, when X, Y, and Z are normally distributed. He finds that

(n~2)

3) !v';

Downloaded by [71.74.122.179] at 16:09 08 January 2016

n ( -2-

where y= {

J.laa(J.l22b2 - 2J.l12b

+ J.lll) -1 } -1/2

(J.l2ab - J.l13) 2

'

where EX=EY=EZ=O and J.lll=Ey2, J.l22=EX2, J.laa=EZ2, J.l12=EXY, J.l13 =EYZ, and J.l23=EXZ. We see, then, that y is distributed as (n-l)- 1/2t(n-I), where ten-I) is a random variable with a t-distribution with n-I degrees of freedom. We can obtain the approximate variance of b as follows. Write

W 2~

JJ.laa(J.l22{32 t

2J.l12{3

(J.l2a{3 -

{

+ J.lll) _

I}-1

J.l1a) 2

2!J-23(!J-2313 - !J-13)

189

FITTING OF STRAIGHT LINES

The same estimate has been obtained in an unpublished paper of Tukey by the following consideration. Cov (Z, y-bx)=Cov (Z, a+{JX+v-bX-bu) = ({J - b) Cov (Z, X) is zero if and only if b = {J. Thus, we can estimate {J by the value of b for which the slope of the regression of Z on y - bx is zero. But if one considers the sum of squares of deviations due to regression of Z on y - bx, one sees that n

b

=

LZiYi _'_'~_1_ _ n

L~ZiXi

Downloaded by [71.74.122.179] at 16:09 08 January 2016

j-1

makes this quantity zero. The idea is the same as that of Geary, who also formmally considers the sample covariance of Z with y-bx to obtain his estimate. However, Tukey's method of motivating this estimate by considering the analysis of variance in regression technique is different. In our next section, we shall see what more can be obtained from considering linear relations with errors in both variables in the light of the analysis of variance. 4. USE OF VARIANCE COMPONENTS

4.1. Replication of Observations

Let us now consider the situation in which we have another kind of additional information, namely, where we know that we have N i observations Xij on each of nX/s. If we have the situation in which Yij= Yi+Vij and Xij=Xi+Uiil and if the usual assumptions of independence are made, then one can perform a one-criterion analysis of variance on the x's and the y's, and from this obtain an estimate of (J. The simplest way of describing the procedure is to exhibit the an ova table.

Source

$]

I

2~

t

(Z.-2)2

i_I

II

Cov(x, y) +,8B2

t

(Z. -Z)2

i_I

" (Z. -Z)2 U.2+ ,82B 2 L

III IV V VI

'-l

u.s

CovE(x, y) u.s

Here B is the slope of the regression of x on Z . We see then that (I - IV) / ( is an estimate of B2, (II - V) /

(

is an estimate of {jB2, and

(III - VI) / (

'f

(Zi -

E E

Z)2)

(Zi -

Z)2)

(Zi -

Z)2)

is an estimate of {j2B2, Hence (II- V)!(I-IV), (III- VI)!(II- V), and

• IIII - VI 11 I - IV

193

FITTING OF STRAIGHT LINES

are all estimates of (3, provided that the denominators of these estimates do not approach zero as n-too, i.e., BrfO. In all three situations, replications, grouping, and use of an instrumental variable, we have, from this approach, three different estimates of (3. Let us examine the variance of the two estimates easiest to compute, b1 and bs, to determine when each of them should be used. I shall use the approximate relations (II- V)"-'{3(I-IV) and (III- VI)"-'{32(I-IV) in this comparison. Using 28.4 of [7], we see that (1)

I- V) 1

G

Downloaded by [71.74.122.179] at 16:09 08 January 2016

III - VI) (2) (12 ( . II - V (1)-(2)

~

{J2

= - 2 (12 I _ IV

-

1

(32k 2

k

--

{J2k 2

1

(12 II I _ V1

-

{J2 - (12 I _ Iv

-

(32 - 2{32

+ 2(34 (12 _

1

k2

=

~

k2

I Iv -

f32

>

-

k2

2{J -

k2

{32 -

k2

-

-

{J2k2

(12 I _ IV

k

Cov (II - V I - IV) '

2(32k Cov (II - V III - VI) ' (33k 3

(Cov (II - V I - IV - III '

+ VI))

(1 - (J2) (Cov(II - V I - IV)) '

~

--

2(3 -

k3

2{J(1 - (32)

1

I Iv -

+ 2(34 (12 _

- (32

k2 ~

-(3~2

(12II _ v

2(3

-

~

(32k2

(34k 4

(12 III _ V1

-

--

+-

{J2 - (12 I _ Iv

(12 I _ IV

4

(34k 2

(12 III _ V1

~

k2

k

(12 I II _ V 1

1 -

2

+ (3 (32k -

(12 - - "-' - (12II _ V - IV k2

{J(12 I _ I v

(12 III _ V1

>

0

1

where k is either B2

" L: (Zi -

'2)2

i=l

(in the instrumental variable case),

(in the replication situation), or

(in the grouping situation). We see, then, that b2 is a better estimate of (3 than bdf we know that {J > 1, and that bds better if {J < 1. An idea as to whether or not 1(31 > 1 can be obtained from plotting the observations. In [45] Tukey gives the approximate variance of b: and b2. Since b3 is the geometric mean of b: and bs; the approximate variance of b« can be determined

II

II

194

AMERICAN STATISTICAL ASSOCIATION JOURNAL, MARCH 1959

by 28.4 of [7]. We shall give the approximate variance of b1 ; the changes which . must be made to obtain Var b2 are obvious. I(IlI - 2Ilb1

+ Ib 2) 1

IV(VI - 2Vb1

+ IVb

2 1 )

df(I) + df(IV) Var b1 ~--------------------(I - IV)2 where in case of replications in case of grouping in case of instrumental variables

Downloaded by [71.74.122.179] at 16:09 08 January 2016

and N - n

df(IV)

=

j

in case of replications

n - r

in case of grouping

n-2

in case of instrumental variables.

This approxination is good for large df(I). Hence it will not be too good in the usual cases of grouping, where r is very small, or when one has an instrumental variable. For this reason, the estimated variances via this method are not given . in Table 200. 5. THE BERKSON MODEL

SO far, we have considered the case in which either (x, y) was a random pair or the special case where (x, y) was chosen so that the grouping method was (at least approximately) applicable. There is, however, another model, due to Berkson, [5], (of, also [22] and [28]), wherein instead of trying to observe a given Xi but actually observing Xi=Xi+Ui, we fix our xi's and observe Yi for each fixed Xi. This process of fixing one's xi's can be done quite easily in the laboratory sciences, where, for example, if one wished to estimate an Ohm's law constant (e.g., a resistance), one could fix the xis by setting the dial of the ammeter (presumably the cause of the errors in observing Xi) at predetermined settings. Then, for each fixed x (e.g., for each fixed current reading), there are a number of X's which could have given rise to the particular x which is observed; also for each X there is a probability that the observed fixed x is an observation on that X with error u. The X's are now random variables distributed about the fixed x with error u, i.e., X =x+u where U is independent of x (and not of X). We now observe y=a+l3x+l3u+v and here both l3u and v are independent of x. Hence we have the situation in which our relation is y = a+l3x+w where w=iJu+v and w is independent of z. If we assume Eu=Ev=O, then Ey=a+iJx. x is now a fixed number rather than a random variable, and we know that for this situation the least squares estimate of 13 is

195

FITTING OF STRAIGHT LINES n

L: (Xi b=

X)(Yi - y)

_'_'~_l

_ n

L: (Xi -

X)2

Downloaded by [71.74.122.179] at 16:09 08 January 2016

i=l

This technique works not only in the structure-function situation, but also in the regression situation, for the distinction in errors in Y makes no difference in this case. The real significance of the Berkson model lies not in the estimator of {3 but rather in the implications of the model to the design of experiments intended to yield data from which {3 may be estimated. In contrast to the problems in estimating {3 described above when (x, y) is a random pair, we see that if our physicist can fix his x's, the statistician has no problem in estimating the linear relation of interest. 6. ESTIMATION VIA CUMULANTS

Consider the homogeneous linear relation {3IX + {32 y = 0 where f3 = - f3r/ {32 and X and Yare random variables whose expectation is zero. That is, let us consider a structural relation where the intercept is zero. Geary [12, 13] noticed that since Y =(3X, the bivariate cumulant of X and Y of order CI, c2+1, namely K(CI, c2+1), was equal to (3K(CI+l, C2). Also, if CI, C2>0, K(CI, C2), the cumulant of the distribution of the x's and V's, is equal to K(Cl, C2). This is evident from the following properties of bivariate cumulants (cf. Kendall [23], Kaplan [20], and Lindley [27]): (a) the cumulant of a sum of independent random variables is the sum of the cumulants of the variables, and (b) the bivariate cumulant of any order CI+C2 (where both CI, C2>0) of.Independent random variables is zero. Since Cov(u, X) =Cov(v, Y) =Cov(u, v) =0, K(CI, C2) =K(CI, C2)+ [(CI, C2)-th cumulant of X and u]+ [(CI, C2)-th cumulant of Y and v] + [(CI, c2)-th cumulant of u and v] = K(CI, C2) +0+0+0 = K(CI, C2). Hence, since k(cI, C2), the sample k~statistic, is an unbiased consistent estimate of K(CI, C2), we see that

is a consistent estimate of (3 if k(ci + 1, C2) does not approach zero. This estimate has not used any additional information and hence by the identifiability result we know that in the case of normality it must fail. This is certainly correct, for in the normal distribution, all cumulants of degree three or higher are zero, and since CI>O, C2>O, and cumulants of order cI+c2+1 are used, we cannot estimate {3 by this method in the normal case', Another problem in using this method is that of what order cumulants to use, for this method provides us, in the, non-normal case, with an infinity of estimates based on the different orders of the cumulants used. Geary suggests that cumulants of lowest order be used because of ease of computation. Even so,

Downloaded by [71.74.122.179] at 16:09 08 January 2016

196

AMERICAN STATISTICAL ASSOCIATION JOURNAL, MARCH 1959

one should know something about the shape of the joint distribution of x and y, for all odd cumulants of a symmetric distribution are zero, and hence cannot be used to estimate {3. A better measure of what order eumulants to be used is the variance of the estimates based on different orders. To be on the safe side, fourth order cumulants should be used to estimate {3 if nothing is known about the joint distribution of x and y, since the distribution may be symmetric and then use of third order cumulants will not yield an estimate of {3. However, one should note that inaccuracy in estimation of cumulants generally increases rapidly with order, and, coupled with the fact that this general method of estimation breaks down in the normal case, one is not very likely to use it to estimate {3. One method of improving estimation via cumulants is to pool estimates of (3 based on different values of CI, C2. The linear combination of estimates S(CI, C2) and S(CI*, C2*) which has minimum variance is the combination as(cI, C2) +(I-a)S(cI*, C2*)' where a=

V(S(CI*, C2*)) - Cov (S(CI, C2), S(CI*, C2*)) V(S(Cl, C2) - S(Cl*, C2*))

To facilitate this averaging, the asymptotic variance of S(CI, C2) and the asymptotic covariance of S(CI, C2) and S(CI*, C2*) should be determined. But, using the multivariate extension of 28.4 of [7], V[

+ I)J ~ (32 [V(k(Cl' C2 + 1)) + V(k(ci + 1, C2)) ---+ 1, C2) K C2 + 1) K + 1, C2) _ 2C(k(cI, C2 + 1), k(ci + 1, C2))J, K(Cl, C2 + I)K(CI + 1, C2)

k (CI, C2 k(ci

2(CI,

2(CI

and C [k(CI, C2 + 1), k(cI*, C2* + I)J k(ci + 1, C2) k(cI* + 1, C2*) ~

+ 1), k(cI*, C2* + 1)) + C(k(CI + 1, C2), k(CI* + 1, C2*)) --------+ I)K(CI*, C2* + 1) K(Cl + 1, C2)K(CI* + 1, C2*) C(k(cI, C2 + 1), k(CI* + 1, C2*)) _ C(k(CI + 1, C2), k(CI*, C2* + 1))J, K(CI, C2 + 1)K(CI* + 1, C2*) K(CI + 1, C2)K(CI*, C2* + 1)

(32 [

_

C(k (CI' C2 K(el, C2

and each of these variances and covariances can be determined using the procedure of [20]. Estimation of (3 via moments rather than cumulants has also been considered. Since the ideas behind and the problems besetting this method are similar to those presented above, I shall only refer the reader to [9] and [38] for a fuller discussion of these estimates. 7. ESTIMATION IN IDENTIFIABLE CASES

An entirely different approach to the problem of estimating {3 in the linear structural relation is taken by Neyman [33]. He first rewrites th-e equation

197

FITTING OF STRAIGHT LINES

Y =a+,8X in polar coordinates as Xcos 0*+ Y sin 0* =P where 0* =tan- ,8 and -1r/2