Neural Networks Fundamentals

The Perceptron

The simplest neural network unit — computes a weighted sum and applies a step function:

A single perceptron can only learn linearly separable functions. 它無法學會 XOR — 這個限制推動了 multi-layer networks 的發展。

Multi-Layer Perceptron (MLP)

An MLP stacks multiple layers of neurons with nonlinear activations:

where $\sigma$ is a nonlinear activation function.

Why nonlinearity is essential: Without it, stacking layers is pointless — $\mathbf{W}_2(\mathbf{W}_1\mathbf{x}) = \mathbf{W}'\mathbf{x}$（just a single linear transformation）。Nonlinearity 讓 network 能 approximate 任意複雜的函數。

Activation Functions

Comparison Table

ReLU and Its Variants

ReLU: $\max(0, x)$

• Pros: 計算高效、mitigates vanishing gradient（gradient = 1 for positive values）、promotes sparsity（很多 neurons output 0）
• Cons: Dying ReLU — 如果某 neuron 的 input 永遠為負 → gradient = 0 → 永遠不更新 → 這個 neuron「死了」

Leaky ReLU: $\max(\alpha x, x)$, $\alpha \approx 0.01$

解決 dying ReLU — negative 側有小的 gradient（$\alpha$），讓 neuron 還有機會 recover。

GELU (Gaussian Error Linear Unit): $x \cdot \Phi(x)$

Smooth approximation of ReLU。在 Transformers（BERT, GPT）中幾乎是 standard。比 ReLU smooth → training 更穩定。

Sigmoid: Why Not in Hidden Layers

Sigmoid's maximum derivative is 0.25. After $L$ layers:

10 layers → $0.25^{10} \approx 10^{-6}$ → vanishing gradients。所以 sigmoid 只在 output layer（binary classification）或 gating mechanisms（LSTM forget gate）中使用，不在 hidden layers。

Softmax: Details

Converts logits → probability distribution（all values > 0, sum = 1）。

Choosing Activation Functions

Universal Approximation Theorem

A feedforward network with a single hidden layer and a finite number of neurons can approximate any continuous function on a compact subset of $\mathbb{R}^n$.

面試中的 key nuance — 定理保證存在性但不保證：

• 需要多少 neurons（可能需要 exponentially many）
• Gradient descent 能不能找到正確的 weights
• Network 能不能泛化到 unseen data

這就是為什麼 depth matters — deep networks 可以用 exponentially fewer parameters 表示某些函數。

Vanishing and Exploding Gradients

In a deep network with $L$ layers, gradients involve a product of $L$ factors:

每個 factor 涉及 weight matrix 和 activation derivative。如果 factors consistently < 1 → gradients vanish exponentially。Consistently > 1 → gradients explode。

Solutions

Weight Initialization

Proper initialization keeps the variance of activations and gradients stable across layers — 太大會 explode，太小會 vanish。

Xavier / Glorot Initialization

Designed for sigmoid and tanh:

確保每層 output 的 variance ≈ input 的 variance → signal 不會隨 depth shrink or grow。

He / Kaiming Initialization

Designed for ReLU:

ReLU 把一半的值 zero out → output variance halved。Factor of 2 compensates for this。

Why Initialization Matters

Batch Normalization

Normalize the input to each layer across the mini-batch:

where $\mu_B$, $\sigma_B^2$ are mini-batch statistics, $\gamma$ and $\beta$ are learnable parameters.

Why BatchNorm Works

Train vs Inference

BatchNorm vs LayerNorm

LayerNorm 在 Transformers 中是 standard — 因為 sequence length 可變，batch statistics 不可靠。

Dropout

During training, randomly set each neuron's output to zero with probability $p$:

Scaling factor $\frac{1}{1-p}$ = inverted dropout — 確保 expected output 不變，inference 時不需要調整。

Why Dropout Works

• Prevents co-adaptation: Each neuron must learn to be useful independently
• Implicit ensemble: Training with dropout ≈ averaging exponentially many sub-networks
• Regularization: 和 L2 regularization 有理論上的等價關係（Gaussian dropout ≈ L2）

Dropout in Practice

Other Regularization Techniques

Hands-on: Neural Networks in PyTorch

MLP

import torch
import torch.nn as nn

class MLP(nn.Module):
    def __init__(self, input_dim, hidden_dim, output_dim, dropout=0.5):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(input_dim, hidden_dim),
            nn.BatchNorm1d(hidden_dim),
            nn.ReLU(),
            nn.Dropout(dropout),
            nn.Linear(hidden_dim, hidden_dim),
            nn.BatchNorm1d(hidden_dim),
            nn.ReLU(),
            nn.Dropout(dropout),
            nn.Linear(hidden_dim, output_dim),
        )

    def forward(self, x):
        return self.net(x)

model = MLP(input_dim=784, hidden_dim=256, output_dim=10)
# Note: no activation after last layer — CrossEntropyLoss includes softmax

Training Loop

import torch.optim as optim

criterion = nn.CrossEntropyLoss()  # includes softmax internally
optimizer = optim.Adam(model.parameters(), lr=1e-3, weight_decay=1e-4)

for epoch in range(num_epochs):
    model.train()  # enable dropout + batch norm training mode
    for X_batch, y_batch in train_loader:
        logits = model(X_batch)
        loss = criterion(logits, y_batch)

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

    model.eval()  # disable dropout + use running stats for batch norm
    with torch.no_grad():
        val_logits = model(X_val)
        val_loss = criterion(val_logits, y_val)

Initialization

# He initialization for ReLU layers
def init_weights(m):
    if isinstance(m, nn.Linear):
        nn.init.kaiming_normal_(m.weight, nonlinearity="relu")
        nn.init.zeros_(m.bias)

model.apply(init_weights)

Real-World Use Cases

Case 1: 信用卡詐欺偵測 — MLP vs Traditional ML

面試 follow-up：「什麼時候 NN 比 GBM 好？」— 當 features 是 raw/unstructured（images, text, sequences）或資料量非常大（> 10M rows, NN 的 capacity 才能被利用）。對 moderate-size tabular data，GBM 幾乎總是更好。

Case 2: 推薦系統 — Embedding Layers

Neural networks 在推薦系統中的核心貢獻是 embedding layers — 把 high-cardinality categorical features（user ID, item ID）映射到 dense low-dimensional vectors：

class RecModel(nn.Module):
    def __init__(self, n_users, n_items, embed_dim=64):
        super().__init__()
        self.user_embed = nn.Embedding(n_users, embed_dim)
        self.item_embed = nn.Embedding(n_items, embed_dim)
        self.fc = nn.Sequential(
            nn.Linear(embed_dim * 2, 128),
            nn.ReLU(),
            nn.Linear(128, 1),
        )

    def forward(self, user_ids, item_ids):
        u = self.user_embed(user_ids)
        i = self.item_embed(item_ids)
        return self.fc(torch.cat([u, i], dim=1)).squeeze()

Embedding layer 本質上是 a lookup table — 但 weights 透過 backprop 學習，每個 user/item 得到一個 meaningful vector。

Case 3: NLP — From Bag-of-Words to Embeddings

NN 的 word embeddings（Word2Vec → BERT → GPT）revolutionized NLP：

• One-hot: 10K vocab → 10K sparse vector（no semantic info）
• Word2Vec embedding: 10K vocab → 300-dim dense vector（king - man + woman ≈ queen）
• BERT contextual embedding: Same word 在不同 context 中有不同 embedding

面試中知道這個 progression 很重要 — 從 sparse representation 到 learned dense embeddings 是 neural network 在 NLP 中的核心貢獻。

Interview Signals

What interviewers listen for:

• 你知道為什麼需要 nonlinearity（沒有 → stacking layers 等於 single linear layer）
• 你能解釋 vanishing gradient 的原因和解法（ReLU, initialization, skip connections）
• 你知道 Xavier vs He initialization 各自對應什麼 activation
• 你能比較 BatchNorm vs LayerNorm 的適用場景
• 你理解 dropout 的直覺（implicit ensemble, prevent co-adaptation）

Practice

Flashcards

Quiz