Feature Engineering

What You Should Understand

• 知道不同 encoding 方法的適用場景和陷阱
• 理解 scaling 對不同 model 的影響（哪些需要、哪些不需要）
• 能辨認 missing data 的機制（MCAR/MAR/MNAR）並選擇正確的 imputation 策略
• 掌握 feature selection 的三大類方法和 tradeoffs
• 能根據 domain knowledge 設計 time-based、aggregation、interaction features

Categorical Encoding

One-Hot Encoding

Each category becomes a binary column:

# city: [NYC, SF, LA] → city_NYC, city_SF, city_LA
pd.get_dummies(df, columns=["city"], drop_first=True)
# drop_first=True: avoid dummy variable trap (perfect multicollinearity)

Label Encoding

Map each category to an integer: NYC→0, SF→1, LA→2.

只在 tree-based models 中使用 — trees 只做 threshold splits（$x \leq 1$），不在意數字大小的 ordinal meaning。Linear models 會被假的 ordinal 關係誤導。

Target Encoding (Mean Encoding)

Replace category with the mean of the target for that category:

# city → average house price in that city
city_mean = df.groupby("city")["price"].mean()
df["city_encoded"] = df["city"].map(city_mean)

Mitigating leakage:

• Leave-one-out: Calculate mean excluding the current row
• K-Fold: Calculate mean using only out-of-fold data（和 CV 一樣的概念）
• Smoothing: Blend category mean with global mean（Bayesian shrinkage）

$m$ 是 smoothing parameter。Category 的 sample size $n_c$ 小時 → 更靠近 global mean（避免 noisy estimate）。CatBoost 的 ordered target encoding 就是這個思路的延伸。

Hash Encoding (Feature Hashing)

Map categories to a fixed-size vector via hash function:

from sklearn.feature_extraction import FeatureHasher

hasher = FeatureHasher(n_features=100, input_type="string")
X_hashed = hasher.transform(df["city"])
# 10K unique cities → fixed 100 columns (with hash collisions)

適合 ultra-high cardinality（百萬 level 的 user/item IDs）。缺點：hash collision 會混淆不同 categories，不可逆（無法從 hash 推回原始 category）。

Encoding Selection Guide

Feature Scaling

Methods

Which Models Need Scaling?

Missing Data

Missing Mechanisms

Imputation Strategies

from sklearn.impute import SimpleImputer, KNNImputer
from sklearn.experimental import enable_iterative_imputer
from sklearn.impute import IterativeImputer

# Simple: mean for numeric, most_frequent for categorical
num_imputer = SimpleImputer(strategy="median")
cat_imputer = SimpleImputer(strategy="most_frequent")

# KNN: uses nearest neighbors
knn_imputer = KNNImputer(n_neighbors=5)

# MICE: iterative model-based
mice_imputer = IterativeImputer(max_iter=10, random_state=42)

Missing as a Feature

有時候 missingness 本身就是有用的 signal：

• 「裝修年份」missing → 沒裝修過 → 房子可能比較舊
• 「收入」missing → 可能是高收入不想透露
• XGBoost/LightGBM 原生支援 missing values — 自動學習 default split direction

import numpy as np

# Add missing indicator before imputing
df["income_missing"] = df["income"].isna().astype(int)
df["income"] = df["income"].fillna(df["income"].median())
# Model can learn from both the imputed value AND the missingness pattern

Feature Selection

Three Paradigms

Filter Methods

from sklearn.feature_selection import mutual_info_classif, f_classif

# Mutual Information (non-linear relationships)
mi_scores = mutual_info_classif(X, y)

# ANOVA F-test (linear relationships, for classification)
f_scores, p_values = f_classif(X, y)

# Correlation with target (for regression)
correlations = X.corrwith(y).abs().sort_values(ascending=False)

Filter methods 的限制：每次只看一個 feature 和 target 的關係，忽略 feature interactions。Feature A 和 B 各自和 target 無關，但 A×B 可能非常有用（例如 XOR pattern）。

Wrapper Methods

Recursive Feature Elimination (RFE):

from sklearn.feature_selection import RFE
from sklearn.ensemble import RandomForestClassifier

rfe = RFE(
    estimator=RandomForestClassifier(n_estimators=100),
    n_features_to_select=20,
    step=5,  # remove 5 features per iteration
)
rfe.fit(X, y)
selected = X.columns[rfe.support_]

每次移除最不重要的 features → 重新訓練 → 再移除 → 直到剩下目標數量。缺點：非常慢（每 step 都要重新訓練）。

Embedded Methods

from sklearn.linear_model import LassoCV

# Lasso automatically zeros out unimportant features
lasso = LassoCV(cv=5).fit(X, y)
selected = X.columns[lasso.coef_ != 0]
print(f"Selected {len(selected)} / {X.shape[1]} features")

# Tree-based importance
from sklearn.ensemble import RandomForestClassifier
rf = RandomForestClassifier(n_estimators=100).fit(X, y)
important = X.columns[rf.feature_importances_ > 0.01]

Feature Transformation

Numeric Transformations

from sklearn.preprocessing import PowerTransformer

# Yeo-Johnson: auto power transform (works with negatives)
pt = PowerTransformer(method="yeo-johnson")
X_transformed = pt.fit_transform(X)
# Makes features more Gaussian-like

Feature Interaction

Explicitly create interaction features when domain knowledge suggests it：

# Domain knowledge: area × floor_count = total living space
df["total_space"] = df["area"] * df["floor_count"]

# Polynomial interactions (auto-generate)
from sklearn.preprocessing import PolynomialFeatures
poly = PolynomialFeatures(degree=2, interaction_only=True, include_bias=False)
X_interactions = poly.fit_transform(X)
# Warning: p features → p + p*(p-1)/2 columns → can explode

Tree-based models 自動學 interactions（nested splits = implicit interaction），但 linear models 需要 explicitly add them。

Domain-Specific Features

Time-Based Features

# From a datetime column
df["hour"] = df["timestamp"].dt.hour
df["day_of_week"] = df["timestamp"].dt.dayofweek
df["is_weekend"] = df["day_of_week"].isin([5, 6]).astype(int)
df["month"] = df["timestamp"].dt.month
df["is_holiday"] = df["date"].isin(holiday_dates).astype(int)

# Cyclical encoding for periodic features
df["hour_sin"] = np.sin(2 * np.pi * df["hour"] / 24)
df["hour_cos"] = np.cos(2 * np.pi * df["hour"] / 24)
# hour 23 and hour 0 are "close" — sin/cos captures this, label encoding doesn't

Aggregation Features

# User-level aggregations from transaction data
user_agg = transactions.groupby("user_id").agg(
    total_spend=("amount", "sum"),
    avg_spend=("amount", "mean"),
    max_spend=("amount", "max"),
    transaction_count=("amount", "count"),
    unique_merchants=("merchant_id", "nunique"),
    days_since_last=("date", lambda x: (today - x.max()).days),
)

Lag Features (Time Series)

# Previous values as features
df["sales_lag_1"] = df.groupby("store_id")["sales"].shift(1)
df["sales_lag_7"] = df.groupby("store_id")["sales"].shift(7)
df["sales_rolling_7d_mean"] = (
    df.groupby("store_id")["sales"]
    .transform(lambda x: x.shift(1).rolling(7).mean())
)
# shift(1) → avoid leaking current value

Real-World Use Cases

Case 1: 信用卡詐欺偵測

面試 follow-up：「哪些 features 最重要？」— 通常 velocity features（每小時交易次數）和 ratio features（金額 / 歷史平均）最有 signal。Domain knowledge（和 fraud analyst 合作）比 blind feature generation 有效得多。

Case 2: 房價預測

Case 3: 客戶分群 / Churn Prediction

Feature Engineering Pipeline

from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.impute import SimpleImputer
from sklearn.ensemble import GradientBoostingClassifier

numeric_features = ["age", "income", "transaction_count"]
categorical_features = ["city", "product_type"]

preprocessor = ColumnTransformer([
    ("num", Pipeline([
        ("imputer", SimpleImputer(strategy="median")),
        ("scaler", StandardScaler()),
    ]), numeric_features),
    ("cat", Pipeline([
        ("imputer", SimpleImputer(strategy="most_frequent")),
        ("encoder", OneHotEncoder(handle_unknown="ignore")),
    ]), categorical_features),
])

model = Pipeline([
    ("preprocessor", preprocessor),
    ("classifier", GradientBoostingClassifier()),
])
# Entire pipeline: impute → scale/encode → train
# CV-safe: preprocessing happens inside each fold

Interview Signals

What interviewers listen for:

• 你能根據 model 類型選擇正確的 encoding 和 scaling
• 你知道 target encoding 的 leakage risk 和 mitigation
• 你能辨認 missing data mechanism 並選擇正確的 imputation
• 你會結合 domain knowledge 做 meaningful features（不只是 blind polynomial）
• 你知道 feature engineering pipeline 要在 CV 內部做（avoid leakage）

Practice

Flashcards

Quiz