Additive Regression Models - Michigan State University

Lecture 16: Generalized Additive Models

Regression III: Advanced Methods Bill Jacoby Michigan State University

http://polisci.msu.edu/jacoby/icpsr/regress3

Goals of the Lecture • Introduce Additive Models – Explain how they extend from simple nonparametric regression (i.e., local polynomial regression) – Discuss estimation using backfitting – Explain how to interpret their results • Conclude with some examples of Additive Models applied to real social science data

2

Limitations of the Multiple Nonparametric Models • Recall that the general nonparametric model (both the lowess smooth and the smoothing spline) takes the following form:

• As we see here, the multiple nonparametric model allows all possible interactions between the independent variables in their effects on Y—we specify a jointly conditional functional form • This model is ideal under the following circumstances: 1. There are no more than two predictors 2. The pattern of nonlinearity is complicated and thus cannot be easily modelled with a simple transformation or polynomial regression 3. The sample size is sufficiently large 3

Limitations of the Multiple Nonparametric Models (2) • The general nonparametric model becomes impossible to interpret and unstable as we add more explanatory variables, however 1. For example, in the lowess case, as the number of variables increases, the window span must become wider in order to ensure that each local regression has enough cases This process can create significant bias (the curve becomes too smooth) 2. It is impossible to interpret general nonparametric regression when there are more than two variables— there are no coefficients, and we cannot graph effects more than three dimensions • These limitations lead us to the Additive Models

4

Additive Regression Models • Additive regression models essentially apply local regression to low dimensional projections of the data • The nonparametric additive regression model is

The fi are arbitrary functions estimated from the data; the errors ε are assumed to have constant variance and a mean of 0 • Additive models create an estimate of the regression surface by a combination of a collection of onedimensional functions • The estimated functions fi are the analogues of the coefficients in linear regression

5

Additive Regression Models (2) • The assumption that the contribution of each covariate is additive is analogous to the assumption in linear regression that each component is estimated separately • Recall that the linear regression model is

where the Bj represent linear effects • For the additive model we model Y as an additive combination of arbitrary functions of the Xs

• The fj represent arbitrary functions that can be estimated by lowess or smoothing splines 6

Additive Regression Models (3) • Now comes the question: How do we find these arbitrary functions? • If the X’s were completely independent—which will not be the case—we could simply estimate each functional form using a nonparametric regression of Y on each of the X’s separately – Similarly in linear regression when the X’s are completely uncorrelated the partial regression slopes are identical to the marginal regression slopes • Since the X’s are related, however, we need to proceed in another way, in effect removing the effects of other predictors—which are unknown before we begin • We use a procedure called backfitting to find each curve, controlling for the effects of the others 7

Estimation and Backfitting • Suppose that we had a two predictor additive model:

• If we unrealistically knew the partial regression function f2 but not f1 we could rearrange the equation in order to solve for f1

• In other words, smoothing Yi-f2(xi2) against xi1 produces an estimate of α+f1(xi1). • Simply put, knowing one function allows us to find the other—in the real world, however we don’t know either so we must proceed initially with estimates 8

Estimation and Backfitting (2) 1. We start by expressing the variables in mean deviation form so that the partial regressions sum to zero, thus eliminating the individual intercepts 2. We then take preliminary estimates of each function from a least-squares regression of Y on the X’s

3. These estimates are then used as step (0) in an iterative estimation process

4. We then find the partial residuals for X1, which removes Y from its linear relationship to X2 but retains the relationship between Y and X1 9

Estimation and Backfitting (3) The partial residuals for X1 are then

5. The same procedure in step 4 is done for X2 6. Next we smooth these partial residuals against their respective X’s, providing a new estimate of f

where S is the (n × n) smoother transformation matrix for Xj that depends only on the configuration of Xij for the jth predictor 10

Estimation and Backfitting (4) • This process of finding new estimates of the functions by smoothing the partial residuals is reiterated until the partial functions converge – That is, when the estimates of the smooth functions stabilize from one iteration to the next we stop • When this process is done, we obtain estimates of sj(Xij) for every value of Xj • More importantly, we will have reduced a multiple regression to a series of two-dimensional partial regression problems, making interpretation easy: – Since each partial regression is only two-dimensional, the functional forms can be plotted on two-dimensional plots showing the partial effects of each Xj on Y – In other words, perspective plots are no longer necessary unless we include an interaction between two smoother terms 11

Interpreting the Effects • A plot of of Xj versus sj(Xj) shows the relationship between Xj and Y holding constant the other variables in the model • Since Y is expressed in mean deviation form, the smooth term sj(Xj) is also centered and thus each plot represents how Y changes relative to its mean with changes in X • Interpreting the scale of the graphs then becomes easy: – The value of 0 on the Y-axis is the mean of Y – As the line moves away from 0 in a negative direction we subtract the distance from the mean when determining the fitted value. For example, if the mean is 45, and for a particular X-value (say x=15) the curve is at sj(Xj)=4, this means the fitted value of Y controlling for all other explanatory variables is 45+4=49. – If there are several nonparametric relationships, we can add together the effects on the two graphs for any particular observation to find its fitted value of Y 12

Additive Regression Models in R: Example: Canadian prestige data • Here we use the Canadian Prestige data to fit an additive model to prestige regressed on income and occupation • In R we use the gam function (for generalized additive models) that is found in mgcv package – The gam function in mgcv fits only smoothing splines (local polynomial regression can be done in S-PLUS) – The formula takes the same form as the glm function except now we have the option of having parametric terms and smoothed estimates – Smooths will be fit to any variable specified with the s(variable) argument • The simple R-script is as follows:

13

Additive Regression Models in R: Example: Canadian prestige data (2) • The summary function returns tests for each smooth, the degrees of freedom for each smooth, and an adjusted Rsquare for the model. The deviance can be obtained from the deviance(model) command

14

Additive Regression Models in R: Example: Canadian prestige data (3) • Again, as with other nonparametric models, we have no slope parameters to investigate (we do have an intercept, however) • A plot of the regression surface is necessary

15

Additive Regression Models in R: Example: Canadian prestige data (4)

80 60

ige Prest

Additive Model: • We can see the nonlinear relationship for both education and Income with Prestige but there is no interaction between them—i.e., the slope for income is the same at every value of education • We can compare this model to the general nonparametric regression model

40 20 5000 10000 In 15000 co m e 20000

14

8

12 n tio a 10 uc Ed

25000

16

Additive Regression Models in R: Example: Canadian prestige data (5) General Nonparametric Model: 80

60 40 20

14 12

on

5000 10000

Ed uc ati

Prestige

• This model is quite similar to the additive model, but there are some nuances— particularly in the midrange of income—that are not picked up by the additive model because the X’s do not interact

10

15000 Inco me 20000

8 25000

17

Additive Regression Models in R: Example: Canadian prestige data (6) • Perspective plots can also be made automatically using the persp.gam function. These graphs include a 95% confidence region

80 60 40 20 5000 10000 in c15000 om e 20000

14

8

12 on i t 10 ca u ed

25000

red/green are +/-2 se

18

Additive Regression Models in R: Example: Canadian prestige data (7) • Since the slices of the additive regression in the direction of one predictor (holding the other constant) are parallel, we can graph each partialregression function separately • This is the benefit of the additive model—we can graph as many plots as there are variables, and allowing us to easily visualize the relationships • In other words, a multidimensional regression has been reduced to a series of two-dimensional partial-regression plots • To get these in R:

19

0 10 -20

s(income,3.12)

Additive Regression Models in R: Example: Canadian prestige data (8)

0

5000

10000

15000

20000

25000

14

16

10 0 -20

s(education,3.18)

income

6

8

10

12 education

20

20 10 -20

-10

0

s(education,3.18)

10 0 -10 -20

s(income,3.12)

20

Additive Regression Models in R: Example: Canadian prestige data (9)

0 5000

15000 income

25000

6

8

10

12

education

14

16 21

R-script for previous slide

22

Residual Sum of Squares • As was the case for smoothing splines and lowess smooths, statistical inference and hypothesis testing is based on the residual sum of squares (or deviance in the case of generalized additive models) and the degrees of freedom • The RSS for an additive model is easily defined in the usual manner:

• The approximate degrees of freedom, however, need to be adjusted from the regular nonparametric case, however, because we are no longer specifying a jointly-conditional functional form

23

Degrees of Freedom • Recall that for nonparametric regression, the approximate degrees of freedom are equal to the trace of the smoother matrix (the matrix that projects Y onto Y-hat) • We extend this to the additive model: 1 is subtracted from each df reflecting the constraint that each partial regression function sums to zero (the individual intercept have been removed) • Parametric terms entered in the model each occupy a single degree of freedom as in the linear regression case • The individual degrees of freedom are then combined for a single measure:

1 is added to the final degrees of freedom to account for the overall constant in the model 24

Testing for Linearity • I can compare the linear model of prestige regressed on income and education with the additive model by carrying out an analysis of deviance • I begin by fitting the linear model using the gam function

• Next I want the residual degrees of freedom from the additive model

25

Testing for Linearity (2) • Now I simply calculate the difference in the deviance between the two model relative to the difference in degrees of freedom (difference in df=7.3-2=5) • This gives a Chi-square test for linearity

• The difference between the models is highly statistically significant—the additive model describe the relationship between prestige and education and income much better

26

Testing for Linearity • An anova function written by John Fox (see the R-script for this class) makes the analysis of deviance simpler to implement:

• As we see here, the results are identical to those found on the previous slide 27