Generative Models

What You Should Understand

• 能區分 generative 和 discriminative models 的根本差異
• 理解 autoencoder → VAE 的演進和 VAE 的 latent space 性質
• 知道 GAN 的 min-max game、training challenges（mode collapse, instability）
• 了解 diffusion models 的高層概念（forward/reverse process）
• 能說出 generative models 在 DS 中的實際應用

Generative vs Discriminative

Autoencoders

Architecture

An autoencoder learns a compressed representation of data by training to reconstruct its input:

Bottleneck: $\text{dim}(\mathbf{z}) \ll \text{dim}(\mathbf{x})$ forces the network to learn a compressed representation — only the most important information survives。

Relation to PCA

Linear autoencoder with MSE loss = PCA. Non-linear autoencoder（with ReLU）learns a non-linear manifold — more powerful but harder to interpret.

Use Cases

Autoencoder Limitations

Autoencoder 學的 latent space 是 unstructured — 不同 regions 之間沒有 smooth interpolation，不能 sample meaningful new data。這就是 VAE 要解決的問題。

import torch.nn as nn

class Autoencoder(nn.Module):
    def __init__(self, input_dim=784, latent_dim=32):
        super().__init__()
        self.encoder = nn.Sequential(
            nn.Linear(input_dim, 256), nn.ReLU(),
            nn.Linear(256, latent_dim),
        )
        self.decoder = nn.Sequential(
            nn.Linear(latent_dim, 256), nn.ReLU(),
            nn.Linear(256, input_dim), nn.Sigmoid(),
        )

    def forward(self, x):
        z = self.encoder(x)
        return self.decoder(z)

Variational Autoencoder (VAE)

The Key Idea

VAE adds probabilistic structure to the latent space — encoder outputs a distribution (mean + variance), not a single point:

Training 不只 minimize reconstruction error，還要讓 latent distribution 接近 a prior $p(\mathbf{z}) = \mathcal{N}(0, I)$。

The Loss Function (ELBO)

KL divergence 是 VAE 的 magic — 它讓 latent space 有 structure，使得 sampling 和 interpolation 成為可能。

Reparameterization Trick

Problem: Sampling from $\mathcal{N}(\boldsymbol{\mu}, \boldsymbol{\sigma}^2)$ 是 stochastic operation → cannot backpropagate through it。

Solution: Reparameterize as deterministic function + external noise:

現在 $\mathbf{z}$ 是 $\boldsymbol{\mu}$ 和 $\boldsymbol{\sigma}$ 的 differentiable function → can backpropagate normally。Randomness 被 externalize 到 $\boldsymbol{\epsilon}$。

VAE vs Autoencoder

class VAE(nn.Module):
    def __init__(self, input_dim=784, latent_dim=32):
        super().__init__()
        self.encoder = nn.Sequential(nn.Linear(input_dim, 256), nn.ReLU())
        self.fc_mu = nn.Linear(256, latent_dim)
        self.fc_var = nn.Linear(256, latent_dim)
        self.decoder = nn.Sequential(
            nn.Linear(latent_dim, 256), nn.ReLU(),
            nn.Linear(256, input_dim), nn.Sigmoid(),
        )

    def encode(self, x):
        h = self.encoder(x)
        return self.fc_mu(h), self.fc_var(h)  # μ and log(σ²)

    def reparameterize(self, mu, log_var):
        std = torch.exp(0.5 * log_var)
        eps = torch.randn_like(std)
        return mu + std * eps  # z = μ + σ * ε

    def forward(self, x):
        mu, log_var = self.encode(x)
        z = self.reparameterize(mu, log_var)
        return self.decoder(z), mu, log_var

Generative Adversarial Networks (GANs)

The Min-Max Game

Two networks compete against each other:

• Generator $G$: Takes random noise $\mathbf{z} \sim p(\mathbf{z})$ → produces fake data $G(\mathbf{z})$
• Discriminator $D$: Distinguishes real data from fake — outputs probability of being real

直覺：Generator = 偽鈔製造者，Discriminator = 警察。兩者互相提升 — 最終 Generator 產出的 fake 和 real 無法區分。

Training Dynamics

# GAN training loop (simplified)
for batch in dataloader:
    # 1. Train Discriminator
    real_labels = torch.ones(batch_size)
    fake_labels = torch.zeros(batch_size)

    z = torch.randn(batch_size, latent_dim)
    fake = generator(z).detach()  # don't update G

    d_loss = bce_loss(discriminator(real_data), real_labels) + \
             bce_loss(discriminator(fake), fake_labels)
    d_loss.backward()
    d_optimizer.step()

    # 2. Train Generator
    z = torch.randn(batch_size, latent_dim)
    fake = generator(z)
    g_loss = bce_loss(discriminator(fake), real_labels)  # want D to think fake is real
    g_loss.backward()
    g_optimizer.step()

GAN Training Challenges

GAN Variants

Diffusion Models

High-Level Idea

Diffusion models learn to reverse a gradual noise-adding process:

Forward process (fixed, not learned): Gradually add Gaussian noise to data over $T$ steps until it becomes pure noise:

Reverse process (learned): A neural network learns to denoise step by step:

Training: Network learns to predict the noise $\boldsymbol{\epsilon}$ that was added at each step:

Generation: Start from pure noise $\mathbf{x}_T \sim \mathcal{N}(0, I)$ → iteratively denoise → get $\mathbf{x}_0$。

Why Diffusion Models Won

Key Applications

Model Comparison

Real-World Use Cases

Case 1: 信用卡詐欺 — Anomaly Detection with Autoencoder

Train autoencoder on normal transactions only → high reconstruction error = potential fraud:

# Train on normal data
autoencoder.fit(X_normal)

# At inference: compute reconstruction error
reconstructed = autoencoder(X_test)
errors = ((X_test - reconstructed) ** 2).mean(dim=1)

# High error → anomaly → potential fraud
threshold = np.percentile(errors_on_validation, 99)
is_fraud = errors > threshold

這和 PCA reconstruction error 的思路一樣 — normal data 在 low-dimensional manifold 上 → 可以被 well-reconstructed。Fraud data 偏離 manifold → 高 reconstruction error。

Case 2: Imbalanced Data — Synthetic Minority Oversampling

GAN 或 VAE 可以為 minority class 生成 synthetic samples — 比 SMOTE 更好因為 captures non-linear manifold：

# Train VAE only on fraud transactions
vae = VAE(input_dim=features.shape[1], latent_dim=16)
vae.fit(X_fraud)

# Generate synthetic fraud samples
z = torch.randn(1000, 16)  # sample from latent space
synthetic_fraud = vae.decode(z)
# Augment training data with synthetic fraud → retrain classifier

Case 3: Privacy-Preserving Synthetic Data

用 generative model 產出 synthetic dataset 取代 real data — 保護 privacy：

醫療、金融等 privacy-sensitive 領域越來越多使用 synthetic data 做研究和 model development。

Hands-on: Generative Models in PyTorch

VAE

import torch
import torch.nn as nn
import torch.nn.functional as F

class VAE(nn.Module):
    def __init__(self, input_dim=784, hidden_dim=256, latent_dim=32):
        super().__init__()
        self.encoder = nn.Sequential(nn.Linear(input_dim, hidden_dim), nn.ReLU())
        self.fc_mu = nn.Linear(hidden_dim, latent_dim)
        self.fc_var = nn.Linear(hidden_dim, latent_dim)
        self.decoder = nn.Sequential(
            nn.Linear(latent_dim, hidden_dim), nn.ReLU(),
            nn.Linear(hidden_dim, input_dim), nn.Sigmoid(),
        )

    def encode(self, x):
        h = self.encoder(x)
        return self.fc_mu(h), self.fc_var(h)

    def reparameterize(self, mu, log_var):
        std = torch.exp(0.5 * log_var)
        eps = torch.randn_like(std)
        return mu + std * eps

    def forward(self, x):
        mu, log_var = self.encode(x)
        z = self.reparameterize(mu, log_var)
        return self.decoder(z), mu, log_var

def vae_loss(recon_x, x, mu, log_var):
    recon = F.binary_cross_entropy(recon_x, x, reduction="sum")
    kl = -0.5 * torch.sum(1 + log_var - mu.pow(2) - log_var.exp())
    return recon + kl

Simple GAN

class Generator(nn.Module):
    def __init__(self, latent_dim=100, output_dim=784):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(latent_dim, 256), nn.ReLU(),
            nn.Linear(256, 512), nn.ReLU(),
            nn.Linear(512, output_dim), nn.Tanh(),
        )

    def forward(self, z):
        return self.net(z)

class Discriminator(nn.Module):
    def __init__(self, input_dim=784):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(input_dim, 512), nn.LeakyReLU(0.2),
            nn.Linear(512, 256), nn.LeakyReLU(0.2),
            nn.Linear(256, 1), nn.Sigmoid(),
        )

    def forward(self, x):
        return self.net(x)

Interview Signals

What interviewers listen for:

• 你能區分 generative vs discriminative models
• 你知道 VAE 的 KL loss 為什麼讓 latent space 有 structure
• 你能解釋 GAN 的 min-max game 和 mode collapse 問題
• 你知道 diffusion models 為什麼取代了 GANs
• 你能說出 generative models 在 DS 中的實際應用（anomaly detection, data augmentation, synthetic data）

Practice

Flashcards

Quiz