Loading Curriculum...

Artificial Intelligence (AI) Deep Learning

Autoencoders

Lesson 6/11

Deep Learning â€¢ Representation Learning

Autoencoders

Autoencoders learn an encoder that compresses data into a latent code and a decoder that reconstructs the original input. They are a core tool for dimensionality reduction, anomaly detection, and modern generative models (VAEs, diffusion latents).

Definition

Schema

Formally, an autoencoder is a pair of functions: an encoder EÏ† : X â†’ Z and a decoder DÎ¸ : Z â†’ X. Training minimizes reconstruction error:

L(?, f) = E_{x ~ µ_ref} [ d(x, D?(Ef(x))) ]

Undercomplete AEs (dim(Z) < dim(X)) act like compression. With sufficient capacity and an overcomplete latent, the AE can learn the identity unless we add regularization.

Math

Mathematical Principles

Typical setup: X = R^m, Z = R^n with m > n. A simple one-layer encoder is EÏ†(x) = Ïƒ(Wx + b).

With an L2 reconstruction loss, training becomes least squares:

min_{θ, φ} (1/N) Σ_i || x_i − Dθ(Eφ(x_i)) ||^2

Linear undercomplete autoencoders recover the same subspace as PCA (up to rotation).

Intuition

Interpretation

Think of the latent code as a bottleneck. If the bottleneck is small, the model must discard details and keep only the most useful structure.

Reconstruction error can act as an anomaly score: if a point is unlike the training distribution, the AE often reconstructs it poorly.

Depth helps: deep encoders/decoders can represent complex manifolds more efficiently than shallow ones.

Hands-On

Interactive Simulator

This is a small, visual undercomplete autoencoder demo. For linear AEs, the solution matches PCA: compress to 1D along the dominant direction and reconstruct back into 2D. Toggle denoising to see why DAEs learn robust features.

Simulator

Denoising mode (train on noisy, reconstruct clean)

noise

0.35

outliers

anomaly

Ã—0.55

Recon error is used as an anomaly score. Raise the anomaly slider to be more conservative.

Reading The Plot

Dark dots are original data. Faint dots are corrupted inputs (only in denoising mode). White circles are reconstructions.

When latent=1D, reconstructions are forced onto a single â€œmanifold directionâ€ (PCA-like). Thatâ€™s compression.

Outliers (red) tend to reconstruct poorly, which is why reconstruction error can work as an anomaly score.

Variants

Variations

Objective

min_{Î¸,Ï†} E_x [ d(x, DÎ¸(EÏ†(x))) ] (often L2 reconstruction)

Use Cases

Compression, embeddings, pretraining, anomaly detection (via recon error).

Note

If latent is too large and unregularized, AE can learn identity and be useless.

Autoencoder (AE)

Undercomplete encoder-decoder trained to reconstruct inputs; learns compressed representations.

Denoising AE (DAE)

Corrupt inputs and train to reconstruct clean data; learns robust features.

Sparse AE (SAE)

Add sparsity so only a few latent units activate; encourages parts-based features.

Contractive AE (CAE)

Penalize encoder Jacobian; makes representations stable to small input changes.

Variational AE (VAE)

Latent is a distribution; optimize ELBO (recon + KL). Enables generation by sampling.

MDL AE

Minimum description length framing: compress code + reconstruction error jointly.

Concrete AE

Differentiable feature selection: latent chooses input features via a relaxed categorical distribution.

Where They Show Up

Applications

Dimensionality reduction and embeddings
Anomaly detection via reconstruction error
Image denoising / compression / super-resolution
Information retrieval (semantic hashing)
Drug discovery and molecule generation (VAE variants)
Communication systems: end-to-end learned codecs

Practice

Training Notes

AEs are trained with gradient descent on reconstruction loss, often with regularization (dropout, sparsity, weight decay).

VAEs require careful balance between reconstruction and KL terms (Î²-VAE). Too much KL can cause posterior collapse.

For anomaly detection, always validate thresholds: some AEs can reconstruct anomalies surprisingly well.

Loading Lab...

Deep Learning â€¢ Representation Learning

Autoencoders

Definition

Schema

Formally, an autoencoder is a pair of functions: an encoder EÏ† : X â†’ Z and a decoder DÎ¸ : Z â†’ X. Training minimizes reconstruction error:

L(?, f) = E_{x ~ µ_ref} [ d(x, D?(Ef(x))) ]

Undercomplete AEs (dim(Z) < dim(X)) act like compression. With sufficient capacity and an overcomplete latent, the AE can learn the identity unless we add regularization.

Math

Mathematical Principles

Typical setup: X = R^m, Z = R^n with m > n. A simple one-layer encoder is EÏ†(x) = Ïƒ(Wx + b).

With an L2 reconstruction loss, training becomes least squares:

min_{θ, φ} (1/N) Σ_i || x_i − Dθ(Eφ(x_i)) ||^2

Linear undercomplete autoencoders recover the same subspace as PCA (up to rotation).

Intuition

Interpretation

Think of the latent code as a bottleneck. If the bottleneck is small, the model must discard details and keep only the most useful structure.

Reconstruction error can act as an anomaly score: if a point is unlike the training distribution, the AE often reconstructs it poorly.

Depth helps: deep encoders/decoders can represent complex manifolds more efficiently than shallow ones.

Hands-On

Interactive Simulator

Simulator

Denoising mode (train on noisy, reconstruct clean)

noise

0.35

outliers

anomaly

Ã—0.55

Recon error is used as an anomaly score. Raise the anomaly slider to be more conservative.

Reading The Plot

Dark dots are original data. Faint dots are corrupted inputs (only in denoising mode). White circles are reconstructions.

When latent=1D, reconstructions are forced onto a single â€œmanifold directionâ€ (PCA-like). Thatâ€™s compression.

Outliers (red) tend to reconstruct poorly, which is why reconstruction error can work as an anomaly score.

Variants

Variations

Objective

min_{Î¸,Ï†} E_x [ d(x, DÎ¸(EÏ†(x))) ] (often L2 reconstruction)

Use Cases

Compression, embeddings, pretraining, anomaly detection (via recon error).

Note

If latent is too large and unregularized, AE can learn identity and be useless.

Autoencoder (AE)

Undercomplete encoder-decoder trained to reconstruct inputs; learns compressed representations.

Denoising AE (DAE)

Corrupt inputs and train to reconstruct clean data; learns robust features.

Sparse AE (SAE)

Add sparsity so only a few latent units activate; encourages parts-based features.

Contractive AE (CAE)

Penalize encoder Jacobian; makes representations stable to small input changes.

Variational AE (VAE)

Latent is a distribution; optimize ELBO (recon + KL). Enables generation by sampling.

MDL AE

Minimum description length framing: compress code + reconstruction error jointly.

Concrete AE

Differentiable feature selection: latent chooses input features via a relaxed categorical distribution.

Where They Show Up

Applications

Dimensionality reduction and embeddings
Anomaly detection via reconstruction error
Image denoising / compression / super-resolution
Information retrieval (semantic hashing)
Drug discovery and molecule generation (VAE variants)
Communication systems: end-to-end learned codecs

Practice

Training Notes

AEs are trained with gradient descent on reconstruction loss, often with regularization (dropout, sparsity, weight decay).

VAEs require careful balance between reconstruction and KL terms (Î²-VAE). Too much KL can cause posterior collapse.

For anomaly detection, always validate thresholds: some AEs can reconstruct anomalies surprisingly well.