Lesson 8: Multiple and Non-Linear Regression

Lesson 7 fit a straight line to one predictor and one response. Real processes rarely depend on just one input, and they do not always respond in a straight line. This lesson extends simple linear regression in two directions: adding more predictors (multiple linear regression) and letting the relationship curve (non-linear terms such as quadratic and cubic models). It also covers two questions that follow directly from having a fitted model: how confident is a prediction, and does the data need to be transformed before a model can be trusted at all.

Multiple linear regression

Multiple linear regression extends the simple equation to more than one predictor: Y = b0 + b1X1 + b2X2 + ... + bkXk. A GCHS estimator might predict rehab cost (Y) from a home's square footage (X1) and its age in years (X2) at the same time, rather than one variable alone. Each coefficient, bi, is interpreted as the average change in Y for a one-unit increase in that Xi, holding all the other predictors in the model constant. That "holding constant" phrase is doing real work: it is what separates a multiple regression coefficient from a simple regression slope, and it is only a valid interpretation within the range of data the model was fit on.

Adjusted R-squared

Ordinary R-squared can only stay the same or increase every time another predictor is added to a model, even a predictor that has nothing to do with the response, because least squares can always find some way to use the extra variable to reduce the leftover error slightly. Adjusted R-squared corrects for this by penalizing the count of predictors in the model, so it can go down when a new predictor adds little real explanatory power. Because of that penalty, adjusted R-squared, not plain R-squared, is the number to compare across candidate models that have different numbers of predictors.

Multicollinearity and VIF

Multicollinearity happens when two or more predictors in a multiple regression are themselves strongly correlated with each other, for example a home's square footage and its number of bedrooms. It does not usually hurt the model's overall predictions, but it makes the individual coefficients unstable and their standard errors inflated, so a coefficient's sign, size, or statistical significance can shift a great deal with a small change in the data. The variance inflation factor, VIF, is the standard diagnostic: for predictor j, VIF = 1 / (1 - R²j), where R²j is the R-squared from regressing that predictor on all the other predictors. A VIF of 1 means no inflation at all. Commonly cited rules of thumb flag a VIF above 4 or 5 as worth investigating and a VIF above 10 as a sign of serious multicollinearity; these are rules of thumb, not a fixed statistical law, and different textbooks draw the line in different places.

Worked example: if regressing square footage on the other predictors in the model gives R²j = 0.80, then VIF = 1 / (1 - 0.80) = 1 / 0.20 = 5.0, high enough that many sources would tell you to investigate.

Non-linear models

Not every relationship is a straight line. A quadratic term (X²) or cubic term (X³) can be added to the regression equation to capture curvature, such as rehab cost rising faster for homes at the oldest end of the age range than the straight-line trend for younger homes would predict. Despite the curve in the plotted relationship, a model like Y = b0 + b1X + b2X² is still called "linear regression," because the term describes the equation being linear in its coefficients (the b's), not linear in its plotted shape. Fitting it still uses least squares in the same way as simple regression.

Confidence interval vs. prediction interval

Once a regression model is fit, it can produce two different kinds of interval around a prediction at a given X value, and they answer different questions. A confidence interval (CI) for the mean response estimates a range for the average Y across all cases at that X, for example the average rehab cost across all 1,800-square-foot homes. A prediction interval (PI) estimates a range for one individual future observation, for example the rehab cost of one specific 1,800-square-foot home about to be scheduled. The prediction interval is always wider than the confidence interval at the same X, because it must account for both the uncertainty in estimating the mean line and the additional scatter of an individual case around that line; using a confidence interval where a prediction interval is needed understates the real uncertainty in a single case.

Data transformation and the Box-Cox transformation

When residuals from a regression clearly violate normality or constant variance, transforming the response variable before fitting can sometimes fix both problems at once. The Box-Cox transformation is a family of power transformations indexed by a parameter lambda, and a Box-Cox analysis searches for the lambda value that makes the transformed data closest to normal with stable variance. Common landmark values on that scale: lambda = 1 means no transformation is needed; lambda = 0.5 corresponds to a square root transformation; lambda = 0 corresponds to a natural log transformation (a special case handled separately because the power formula breaks down at exactly zero); and lambda = -1 corresponds to a reciprocal (1/Y) transformation. Once the response is transformed, the model is fit and diagnosed in the transformed units, and predictions are converted back for reporting.