QM.1: Multiple Regression Basics

Predict

In a simple regression, adding a second independent variable can change the coefficient on the first variable. Why would this happen, and what does it tell you about the relationship between the two independent variables?

Model Specification and OLS Estimation

A multiple regression model extends simple linear regression by including two or more independent variables:

$Y_i = \beta_0 + \beta_1 X_{1i} + \beta_2 X_{2i} + \cdots + \beta_k X_{ki} + \epsilon_i$

The ordinary least squares (OLS) method estimates the coefficients $\hat{\beta}_j$ by minimizing the sum of squared residuals:

$\min \sum_{i=1}^{n} \hat{\epsilon}_i^2 = \min \sum_{i=1}^{n} (Y_i - \hat{Y}_i)^2$

Interpreting Coefficients

Each slope coefficient $\hat{\beta}_j$ represents the partial effect of $X_j$ on $Y$ , holding all other independent variables constant. This "ceteris paribus" interpretation is what distinguishes multiple regression from running separate simple regressions.

Example

A model of stock returns: $R_i = 0.02 + 1.2 \cdot R_{market} - 0.3 \cdot \Delta r_{interest} + \epsilon_i$

A 1% increase in market return is associated with a 1.2% increase in stock return, holding interest rate changes constant
A 1 percentage point rise in interest rates is associated with a 0.3% decrease in stock return, holding market return constant

Check Your Understanding

In a regression with 3 independent variables and 100 observations, how many degrees of freedom does the error term have?

OLS Assumptions (Classical Linear Regression Model)

For OLS estimators to be BLUE (Best Linear Unbiased Estimators), the following assumptions must hold:

Assumption	Meaning	Violation
Linearity	$Y$ is a linear function of the independent variables	Misspecified functional form
Independence	Errors are independent of each other	Serial correlation
Homoskedasticity	Error variance is constant: $\text{Var}(\epsilon_i) = \sigma^2$	Heteroskedasticity
Normality	Errors are normally distributed	Unreliable hypothesis tests in small samples
No multicollinearity	Independent variables are not perfectly correlated	Coefficients cannot be estimated
Exogeneity	$E(\epsilon_i	X) = 0$

When these assumptions hold, OLS is efficient (minimum variance among linear unbiased estimators). Violations and their remedies are covered in detail in module QM.3.

Hypothesis Testing and Confidence Intervals

Testing Individual Coefficients (t-test)

To test whether $\beta_j$ is statistically significant:

$t = \frac{\hat{\beta}_j - \beta_{j,0}}{SE(\hat{\beta}_j)}$

with $n - k - 1$ degrees of freedom. The most common null hypothesis is $H_0: \beta_j = 0$ (no effect).

A confidence interval for $\beta_j$ :

$\hat{\beta}_j \pm t_{\alpha/2, \, n-k-1} \cdot SE(\hat{\beta}_j)$

Joint Hypothesis Testing (F-test)

The F-test evaluates whether a group of coefficients is jointly significant. For example, testing $H_0: \beta_1 = \beta_2 = 0$ :

$F = \frac{(SSR_{\text{restricted}} - SSR_{\text{unrestricted}}) / q}{SSR_{\text{unrestricted}} / (n - k - 1)}$

where $q$ is the number of restrictions. A significant F-statistic means at least one of the tested variables has a non-zero effect.

ANOVA Decomposition

$\underbrace{SST}_{\text{Total}} = \underbrace{SSR}_{\text{Regression}} + \underbrace{SSE}_{\text{Error}}$

The regression sum of squares (SSR) captures explained variation; the error sum of squares (SSE) captures unexplained variation.

Try It Yourself

A regression of quarterly GDP growth on inflation rate and unemployment rate yields: GDP = 4.2 - 0.6(Inflation) - 1.1(Unemployment). The SE of the unemployment coefficient is 0.4, and there are 80 observations. Test whether unemployment significantly affects GDP growth at the 5% level.

Multiple Regression Analysis

Jennifer Walsh, CFA, is an equity analyst at Pemberton Capital. She is building a multiple regression model to explain quarterly returns for a technology ETF. Her model uses three independent variables: S&P 500 return (X1), change in the 10-year Treasury yield in percentage points (X2), and change in the VIX index (X3). She estimates the model using 60 quarters of data.

Her regression output is as follows:

Variable	Coefficient	Standard Error	t-Statistic
Intercept	0.008	0.004	2.00
X1 (S&P 500)	1.35	0.15	9.00
X2 (Treasury)	-2.10	1.20	-1.75
X3 (VIX)	-0.04	0.01	-4.00

R-squared = 0.82, F-statistic = 85.1, Durbin-Watson = 1.95

The critical t-value at 5% significance (two-tailed) with 56 degrees of freedom is approximately 2.003.

Problem 1 of 5(basic)

Based on Walsh's regression output, which independent variable is NOT statistically significant at the 5% level?

Explain Back

Explain the difference between a t-test and an F-test in the context of multiple regression. When would you use each one, and what question does each answer?

Reflect

1. Why is the 'holding other variables constant' interpretation of regression coefficients so important for investment analysis?
2. What practical problems might arise if one of the OLS assumptions is violated in a financial model?
3. How does understanding ANOVA decomposition help you assess whether a model is useful?

Key Takeaway

Key Concepts:

Multiple regression: $Y = \beta_0 + \beta_1 X_1 + \cdots + \beta_k X_k + \epsilon$
Each coefficient measures the partial effect holding other variables constant
OLS assumptions: linearity, independence, homoskedasticity, normality, no perfect multicollinearity, exogeneity
t-test: tests individual coefficient significance with $n - k - 1$ df
F-test: tests joint significance of multiple coefficients
ANOVA: SST = SSR + SSE decomposes total variation into explained and unexplained
Degrees of freedom = $n - k - 1$

Connect

The next module covers **Evaluating Model Fit** — how to determine whether your regression model actually explains the variation in the dependent variable well. You will learn about R-squared, adjusted R-squared, and model comparison criteria that build directly on the ANOVA decomposition introduced here.