ML Pipelines & Systems

What You Should Understand

• 能描述一個 end-to-end ML pipeline 的所有 components
• 知道 data drift、concept drift、train-serving skew 的差異和偵測方式
• 能設計 feature store 的 offline/online 架構
• 知道 batch vs online serving 的 tradeoffs
• 能用 systematic framework 回答 ML system design 面試題

End-to-End ML Pipeline

A production ML system has far more components than the model. 點擊任一 component 查看它的細節和連結：

Model 通常是最不容易出問題的 component — most production issues 來自 data quality, feature pipelines, 和 serving infrastructure。

Data Pipeline

Data Collection and Validation

Before any modeling, ensure data reliability:

Data Versioning

Feature Engineering in Production

Feature Store

A centralized system for managing ML features with two key stores:

兩者必須計算完全相同的 features — 任何差異就是 train-serving skew。

Feature Types by Latency

Hybrid pattern（最常見）: Expensive features 用 batch 預計算 → serve 時 combine with real-time context features。

Model Training Pipeline

Training Best Practices

Model Registry

Track model lifecycle:

Registry stores: model artifacts, metadata（training data version, hyperparameters, metrics）, lineage（which data/features/code produced this model）。

Enables reproducibility（recreate any past model）and rollback（revert to previous production model in minutes）。

Model Serving

Serving Patterns

Online Serving Considerations

Model Compression

Monitoring and Drift Detection

Three Levels of Monitoring

Data Drift vs Concept Drift vs Train-Serving Skew

PSI (Population Stability Index)

Retraining Strategy

When to Retrain

Safe Deployment

Offline vs Online Metric Gap

Why They Diverge

ML System Design Interview Framework

面試中回答 ML system design 題的六步框架：

Step 1: Clarify Requirements

Step 2: Define the ML Problem

Step 3: Data and Features

• What data sources are available?
• Feature engineering strategy（static, batch, real-time）
• How to handle missing data, outliers, cold start?

Step 4: Model Selection and Training

• Start with simple baseline → iterate to complex
• Training: data split, validation strategy, hyperparameter tuning
• Offline evaluation: which metrics, on what data?

Step 5: Serving and Deployment

• Batch vs online inference?
• Latency budget and model complexity tradeoff
• A/B testing plan for online validation

Step 6: Monitoring and Iteration

• What metrics to monitor? At what levels?
• How to detect and handle drift?
• Retraining cadence and rollback plan

Real-World System Design Examples

Example 1: Fraud Detection System

Example 2: Recommendation System

Example 3: Search Ranking

Hands-on: ML Pipeline in Python

sklearn Pipeline

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.impute import SimpleImputer
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score

# Pipeline: same transforms at training AND serving time
pipeline = Pipeline([
    ("imputer", SimpleImputer(strategy="median")),
    ("scaler",  StandardScaler()),
    ("model",   RandomForestClassifier(n_estimators=100)),
])

# CV: preprocessing inside each fold (no leakage!)
cv_scores = cross_val_score(pipeline, X, y, cv=5, scoring="accuracy")

# Deploy same pipeline object → no train-serving skew
pipeline.fit(X_train, y_train)
y_pred = pipeline.predict(X_test)

Drift Detection with PSI

import numpy as np

def compute_psi(expected, actual, bins=10):
    """Population Stability Index for drift detection."""
    breakpoints = np.quantile(expected, np.linspace(0, 1, bins + 1))
    breakpoints[0], breakpoints[-1] = -np.inf, np.inf

    exp_pct = np.histogram(expected, bins=breakpoints)[0] / len(expected)
    act_pct = np.histogram(actual, bins=breakpoints)[0] / len(actual)
    exp_pct = np.clip(exp_pct, 1e-4, None)
    act_pct = np.clip(act_pct, 1e-4, None)

    return np.sum((act_pct - exp_pct) * np.log(act_pct / exp_pct))

# Example: monitor feature drift in production
for feature in important_features:
    psi = compute_psi(train_data[feature], production_data[feature])
    if psi > 0.25:
        alert(f"Significant drift in {feature}: PSI={psi:.3f}")

Model Experiment Tracking

import mlflow

mlflow.set_experiment("fraud_detection")

with mlflow.start_run():
    # Log parameters
    mlflow.log_params({"n_estimators": 100, "max_depth": 6, "learning_rate": 0.1})

    # Train model
    model = train_model(X_train, y_train)

    # Log metrics
    mlflow.log_metrics({"auc": 0.95, "precision": 0.82, "recall": 0.78})

    # Log model
    mlflow.sklearn.log_model(model, "model")

    # Later: load model for serving
    # loaded_model = mlflow.sklearn.load_model("runs:/<run_id>/model")

Interview Signals

What interviewers listen for:

• 你有 product 和 system 觀點，不只會調 model parameters
• 你知道 model 上線後才是真正的開始，不是 notebook 分數最好就結束
• 你能把複雜技術翻譯成 stakeholder 聽得懂的決策語言
• 你會主動提到 monitoring、rollback、baseline comparison
• 你能辨別 data drift、concept drift、train-serving skew

Practice

Flashcards

Quiz