Regression

Ordinary Least Squares (OLS)

Linear regression models the relationship between a response $y$ and features $\mathbf{x}$:

The OLS estimator minimizes the sum of squared residuals:

Closed-form solution 存在的條件是 $\mathbf{X}^\top \mathbf{X}$ 可逆。不可逆時（multicollinearity 或 $p > n$），需要 regularization。

The Five OLS Assumptions (Gauss-Markov)

Under these assumptions, OLS is the Best Linear Unbiased Estimator (BLUE):

1. Linearity: $y$ is linear in the parameters（不一定要 linear in features — polynomial regression 仍然 linear in $\beta$）
2. Independence: Observations are independent of each other
3. Homoscedasticity: $\text{Var}(\epsilon_i) = \sigma^2$ is constant across all $i$
4. No perfect multicollinearity: No feature is an exact linear combination of others
5. Exogeneity: $E[\epsilon | \mathbf{X}] = 0$ — errors are uncorrelated with features

Geometric Interpretation

OLS 的幾何直覺：$\hat{\mathbf{y}} = \mathbf{X}\hat{\boldsymbol{\beta}}$ 是 $\mathbf{y}$ 在 column space of $\mathbf{X}$ 上的 orthogonal projection。Residual vector $\mathbf{e} = \mathbf{y} - \hat{\mathbf{y}}$ 和所有 features 正交。

這就是為什麼 $\mathbf{X}^\top \mathbf{e} = 0$ — residuals 和 features 無關是 OLS 的必然結果，不是假設。

MLE Connection

If errors are normally distributed, the MLE for $\boldsymbol{\beta}$ is exactly the OLS estimator. 最小化 squared error = 最大化 Gaussian likelihood。這也是為什麼 MSE 是 regression 的 default loss function。

Goodness of Fit

R-squared

$R^2$ measures the proportion of variance explained by the model. 加更多 features 只會讓 $R^2$ 增加（或不變），即使是 noise features。

Adjusted R-squared

Penalizes model complexity by degrees of freedom. 加入無用 feature 時 adjusted $R^2$ 會下降，比 $R^2$ 更適合做 model comparison。

Other Metrics

Residual Analysis

A well-fitted model should have residuals that:

• Show no pattern when plotted against predicted values（checks linearity + homoscedasticity）
• Are approximately normally distributed（check with Q-Q plot）
• Show no autocorrelation（Durbin-Watson test for time series）

Residual Patterns and Diagnoses

Cook's Distance

Measures how much each observation influences the fitted model:

where $\hat{y}_{j(i)}$ is the prediction without observation $i$. 一般 $D_i > 1$ 或 $D_i > 4/n$ 被認為是 influential point。

面試中的用法：「我會先看 residual plot 找 pattern，用 Cook's distance 找 influential points，然後決定是修正模型還是處理資料品質問題。」

Multicollinearity

When features are highly correlated, OLS coefficients become unstable — small changes in data cause large swings in $\hat{\beta}$.

Detection: Variance Inflation Factor (VIF)

where $R_j^2$ is the $R^2$ from regressing feature $j$ on all other features.

Consequences and Solutions

Consequences: Coefficients are still unbiased but have high variance. Individual p-values 變得不可靠，但 overall prediction 可能仍然不錯。

Solutions:

• Remove one of the correlated features
• Ridge regression — shrinks correlated coefficients, always invertible
• PCA — combine correlated features into orthogonal components
• Domain knowledge — decide which feature is more meaningful

Regularization

Ridge Regression (L2)

Ridge shrinks coefficients toward zero but never sets them exactly to zero. $(\mathbf{X}^\top \mathbf{X} + \lambda \mathbf{I})$ 永遠可逆，完美處理 multicollinearity。

Bayesian interpretation: Gaussian prior on $\boldsymbol{\beta}$ → MAP estimate = Ridge。

Lasso Regression (L1)

Lasso performs automatic feature selection by driving some coefficients to exactly zero.

Why does L1 give exact zeros but L2 doesn't? Geometric intuition：L1 constraint region 是菱形（diamond），corners 在座標軸上。Contour lines 更容易碰到 corner（某些 $\beta_j = 0$）。L2 constraint region 是圓形，沒有 corners。

Bayesian interpretation: Laplace prior on $\boldsymbol{\beta}$ → MAP estimate = Lasso。

Elastic Net

Combines L1 and L2 penalties. 當 features 成群相關時，Lasso 傾向隨機選一個、丟掉其他；Elastic Net 會保留整組。

Choosing Regularization

Tuning $\lambda$

Use cross-validation to find the optimal $\lambda$:

from sklearn.linear_model import RidgeCV, LassoCV

# RidgeCV automatically tunes lambda via cross-validation
ridge = RidgeCV(alphas=[0.01, 0.1, 1.0, 10, 100], cv=5).fit(X, y)
print(f"Best alpha: {ridge.alpha_}")

# LassoCV uses coordinate descent path
lasso = LassoCV(cv=5).fit(X, y)
print(f"Best alpha: {lasso.alpha_}")
print(f"Non-zero features: {(lasso.coef_ != 0).sum()} / {len(lasso.coef_)}")

Polynomial Regression

To capture non-linear relationships while staying within the linear regression framework:

仍然 linear in parameters $\beta$，所以 OLS 完全適用。危險是 overfitting — degree $d$ 接近 $n$ 時會 memorize training data。

Logistic Regression

面試中 logistic regression 常和 linear regression 一起被問到。它不是 regression 但名字裡有 regression：

Key differences from linear regression:

Odds and Log-Odds

Coefficient interpretation：$\beta_j = 0.5$ 意味著 $x_j$ 增加 1 單位，odds 乘以 $e^{0.5} \approx 1.65$（增加 65%）。

Bias-Variance Tradeoff

Use this interactive tool to see how model complexity affects the bias-variance tradeoff:

Regularization 增加 bias（模型更受限）但降低 variance（模型更穩定）。Optimal model 是在 bias 和 variance 之間取得平衡。

Real-World Use Cases

Case 1: 房價預測

Challenge: 房價右偏、positive-only → 用 log transform。

import numpy as np
from sklearn.linear_model import Ridge
from sklearn.model_selection import cross_val_score

# Log-transform target (house prices are log-normal)
y_log = np.log1p(y)  # log(1 + y) to handle y=0

ridge = Ridge(alpha=10)
scores = cross_val_score(ridge, X, y_log, cv=5, scoring="neg_mean_squared_error")
# Predict: exp(model.predict(X_new)) - 1 to get back to original scale

面試 follow-up：

• 「為什麼用 log transform？」— 房價接近 log-normal，transform 後 residuals 更接近常態且 homoscedastic
• 「Feature interpretation 怎麼變？」— coefficient 0.1 on log scale ≈ 10% increase in price
• 「有些 feature 是 categorical 怎麼辦？」— One-hot encoding, 但注意 dummy variable trap（$k$ categories 只需 $k-1$ dummies）

Case 2: 信用卡詐欺偵測

Challenge: Binary outcome → Logistic regression。但 class imbalance 嚴重（0.1% fraud）。

from sklearn.linear_model import LogisticRegression

# class_weight='balanced' adjusts for imbalance
model = LogisticRegression(C=1.0, class_weight="balanced")
# Equivalent to oversampling minority class
# C = inverse regularization strength (smaller C = more regularization)

面試 follow-up：

• 「為什麼不用 linear regression 做 classification？」— 預測值可能超出 [0, 1]，而且 MSE 不適合 binary targets
• 「L1 or L2 regularization？」— L1 如果想做 feature selection（哪些 features 和 fraud 最相關）；L2 如果想 keep all features

Case 3: 客戶消費金額預測

Challenge: Target 有很多零（很多客戶沒消費）→ Two-part model。

Part 1: Logistic regression 預測「是否消費」（$P(y > 0)$） Part 2: Linear regression（on log scale）預測「消費多少」（$E[y \mid y > 0]$）

這比直接對含零的資料跑 regression 更合理 — 因為「不消費」和「消費多少」是兩個不同的決策機制。

Hands-on: Regression in Python

OLS vs Ridge vs Lasso

import numpy as np
from sklearn.linear_model import LinearRegression, Ridge, Lasso
from sklearn.model_selection import cross_val_score

# Correlated features + noise features
n = 200
X1 = np.random.randn(n)
X2 = X1 * 0.9 + np.random.randn(n) * 0.3  # correlated with X1
X3 = np.random.randn(n)
X_noise = np.random.randn(n, 5)             # irrelevant features
X = np.column_stack([X1, X2, X3, X_noise])
y = 3 * X1 + 2 * X3 + np.random.randn(n) * 0.5  # only X1, X3 matter

models = {
    "OLS":   LinearRegression(),
    "Ridge": Ridge(alpha=1.0),
    "Lasso": Lasso(alpha=0.1),
}
for name, model in models.items():
    scores = cross_val_score(model, X, y, cv=5, scoring="r2")
    model.fit(X, y)
    print(f"{name}: CV R²={scores.mean():.3f}, coefs={model.coef_.round(2)}")

# OLS:   unstable coefficients for correlated X1/X2
# Ridge: shrinks all coefficients, but keeps all nonzero
# Lasso: drives noise coefficients to exactly 0 (feature selection)
# True:  [3.00, 0.00, 2.00, 0, 0, 0, 0, 0]

Diagnostics with statsmodels

import statsmodels.api as sm
from statsmodels.stats.outliers_influence import variance_inflation_factor

X_const = sm.add_constant(X)
model = sm.OLS(y, X_const).fit()
print(model.summary())
# Look at: coefficients, p-values, R², F-statistic, Durbin-Watson

# VIF check for multicollinearity
for i in range(X.shape[1]):
    vif = variance_inflation_factor(X, i)
    print(f"Feature {i}: VIF = {vif:.2f}")
# VIF > 10 → multicollinearity problem

Interview Signals

What interviewers listen for:

• 你能說出 OLS 的 5 個假設，並知道 normality 不是其中之一
• 你知道 $R^2$ 的限制，不會只靠它判斷 model quality
• 你能解釋 Ridge vs Lasso 的幾何差異（圓 vs 菱形）和 Bayesian 解釋
• 你知道 multicollinearity 不影響 unbiasedness，只影響 variance
• 你能正確解釋 logistic regression 的 coefficient（log-odds, not probability）

Practice

Flashcards

Quiz