Convolutional Neural Networks

Why Convolutions?

FC layers treat every input pixel independently. For a $224 \times 224 \times 3$ image, a single FC layer with 1000 neurons needs:

就一層就 150M — 不可行。

Convolutions exploit two key inductive biases of images:

The Convolution Operation

A 2D convolution slides a kernel (filter) across input and computes element-wise products:

For multi-channel input (e.g., RGB), each filter spans all input channels:

每個 filter 產出一個 feature map（output channel）。$C_{\text{out}}$ 個 filters → $C_{\text{out}}$ 個 feature maps。

Output Dimensions

For input size $W$, kernel size $K$, padding $P$, stride $S$:

Quick Reference

Parameter Count

For a conv layer with kernel $K \times K$, $C_{\text{in}}$ input channels, $C_{\text{out}}$ output channels:

$+1$ 是 bias per filter。

vs FC: Conv(3→32, 3×3) = 32×(3×3×3+1) = 896 params。等價 FC: 32×32×3×32 = 98,304 params — 109x more。

Key Building Blocks

Stride and Padding

• Stride: Kernel 每次移動的 distance。Stride 2 把 spatial dimensions halve（取代 pooling）
• Padding: 在 input 邊緣加 zeros。沒有 padding → 每層 spatial dimension 縮小 → 限制 depth

Pooling

GAP 在 modern architectures 中取代了最後的 FC layers（ResNet, EfficientNet）— 一個 feature map 一個 class 的 average → 直接接 softmax。

Receptive Field

The region of input that influences a particular output neuron. For stacked $3 \times 3$ convolutions:

每加一層 3×3 → receptive field +2。這就是為什麼 stack small kernels 比 large kernels 好 — 相同 receptive field 但更少 parameters + 更多 nonlinearity（VGGNet 的核心 insight）。

1×1 Convolutions

Operates only across channels at each spatial position — like a per-pixel FC layer:

Uses:

• Bottleneck: Reduce $C_{\text{in}}$ → fewer $C_{\text{out}}$ channels → reduce computation
• Channel mixing: Combine information across feature maps
• Add nonlinearity: 1×1 conv + ReLU = per-pixel nonlinear transform

Popularized by Network-in-Network and used extensively in GoogLeNet/Inception and ResNet bottleneck blocks.

Classic Architectures

面試中能說出每個 architecture 的 key innovation 和它解決的問題是很強的信號：

Architecture Timeline

ResNet: The Game Changer

Introduced skip connections (residual connections):

Network learns the residual $\mathcal{F}(\mathbf{x}) = \mathbf{y} - \mathbf{x}$ instead of the full mapping.

Why it works:

• Gradient highway: Identity shortcut → $\partial \mathbf{y}/\partial \mathbf{x}$ has a +1 term → gradients never vanish through shortcuts
• Easy identity: If extra depth is unnecessary, $\mathcal{F}(\mathbf{x}) = 0$ is easy to learn → $\mathbf{y} = \mathbf{x}$（identity mapping）
• Deeper = better: 前 ResNet 時代，plain networks 加 depth → performance degraded（optimization difficulty, not overfitting）。Skip connections 解決了這個問題

ResNet Variants

Bottleneck 用 1×1 conv 先降 channels → 3×3 conv 在低 channels 做 → 1×1 restore channels → 更少計算量 → 可以更深。

Vision Transformer (ViT)

Vision Transformer 把 image 切成 fixed-size patches，每個 patch 當作一個 token feed 進 standard Transformer：

1. Split image into $16 \times 16$ patches
2. Flatten each patch → linear projection to embedding
3. Add positional embedding
4. Feed into standard Transformer encoder
5. Use [CLS] token output for classification

ViT vs CNN:

Data Augmentation

CNN 的 regularization 主力 — 通過對 training images 做 random transformations來增加 effective dataset size：

from torchvision import transforms

train_transform = transforms.Compose([
    transforms.RandomResizedCrop(224),
    transforms.RandomHorizontalFlip(),
    transforms.ColorJitter(brightness=0.2, contrast=0.2),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
])
# ImageNet mean/std are standard for pre-trained models

Transfer Learning

Pre-trained CNNs (typically on ImageNet) serve as powerful feature extractors.

Why It Works

Lower layers learn universal features (edges, textures, colors) → transfer well across tasks. Higher layers learn task-specific features.

Two Strategies

Transfer Learning Decision Matrix

import torchvision.models as models
import torch.nn as nn

# Load pre-trained ResNet50
model = models.resnet50(pretrained=True)

# Strategy 1: Feature extraction — freeze all conv layers
for param in model.parameters():
    param.requires_grad = False

# Replace classifier head for your task
model.fc = nn.Linear(2048, num_classes)  # only this trains

# Strategy 2: Fine-tune — unfreeze top layers
for param in model.layer4.parameters():
    param.requires_grad = True
# Use smaller lr for pre-trained layers, larger for new head

Real-World Use Cases

Case 1: 信用卡詐欺 — CNN 用在哪？

直覺上信用卡詐欺和 image 無關，但 CNN 可以用在：

• Transaction sequence as image: 把使用者的交易序列（amount × time）轉成 2D representation → 用 1D CNN 抓 temporal patterns
• Signature verification: CNN 比對簽名圖片
• Document fraud: CNN 偵測偽造文件（altered text, inconsistent fonts）

面試 follow-up：「tabular fraud data 用 CNN 嗎？」— 通常不用。Tabular data → GBM > MLP > CNN。CNN 的 inductive bias（translation invariance, locality）在 tabular data 上沒有用。

Case 2: 推薦系統 — Visual Features

CNN 在推薦系統中提取 visual features：

• 電商：商品圖片 → CNN embedding → 用 visual similarity 推薦（「和你看的風格相似的商品」）
• Fashion：CNN 學到 color, pattern, style → visual recommendation
• Real estate：房屋照片 → CNN features 加入 pricing model

Case 3: 醫學影像 — Transfer Learning

Medical imaging 是 transfer learning 的典型場景：

• 資料量小（幾百到幾千張 labeled images）
• 但 ImageNet pre-trained features（edges, textures, shapes）仍然有用
• Fine-tune top layers + heavy augmentation 通常表現很好
• 需要 interpretability → Grad-CAM 顯示 model 關注的區域

Hands-on: CNN in PyTorch

Complete CNN Architecture

import torch
import torch.nn as nn

class SimpleCNN(nn.Module):
    def __init__(self, num_classes=10):
        super().__init__()
        self.features = nn.Sequential(
            # Block 1: [B, 3, 32, 32] → [B, 32, 16, 16]
            nn.Conv2d(3, 32, kernel_size=3, padding=1),
            nn.BatchNorm2d(32),
            nn.ReLU(),
            nn.MaxPool2d(2),
            # Block 2: → [B, 64, 8, 8]
            nn.Conv2d(32, 64, kernel_size=3, padding=1),
            nn.BatchNorm2d(64),
            nn.ReLU(),
            nn.MaxPool2d(2),
            # Block 3: → [B, 128, 4, 4]
            nn.Conv2d(64, 128, kernel_size=3, padding=1),
            nn.BatchNorm2d(128),
            nn.ReLU(),
            nn.MaxPool2d(2),
        )
        self.classifier = nn.Sequential(
            nn.AdaptiveAvgPool2d(1),  # GAP: → [B, 128, 1, 1]
            nn.Flatten(),             # → [B, 128]
            nn.Dropout(0.5),
            nn.Linear(128, num_classes),
        )

    def forward(self, x):
        return self.classifier(self.features(x))

Transfer Learning with ResNet

import torchvision.models as models

# Pre-trained ResNet50
model = models.resnet50(weights="IMAGENET1K_V2")

# Freeze all layers
for param in model.parameters():
    param.requires_grad = False

# Replace classifier
model.fc = nn.Sequential(
    nn.Dropout(0.3),
    nn.Linear(2048, num_classes),
)
# Only model.fc parameters will be updated

Interview Signals

What interviewers listen for:

• 你能計算 output dimension 和 parameter count
• 你知道每個 classic architecture 的 key innovation
• 你能解釋 skip connections 為什麼解決了 deep networks 的 training 問題
• 你知道 transfer learning 的策略和 decision matrix
• 你能比較 CNN vs ViT 的 tradeoffs

Practice

Flashcards

Quiz