Loading Curriculum...

Artificial Intelligence (AI) Deep Learning

OptimizationAlgorithms

Lesson 10/11

Deep Learning • Optimization

Optimization Algorithms

Training a neural network is an optimization problem: find weights that minimize a loss. This page explains the core idea behind gradient descent and why Momentum, RMSProp, and Adam became practical defaults.

Idea

The Core Loop

You pick parameters θ (weights). The model produces predictions. The loss L(θ) measures how wrong those predictions are.

Optimization repeatedly updates parameters to reduce loss:θ ← θ − α · ∇θ L(θ)

Here, α is the learning rate and ∇θ L is the gradient: the direction of steepest increase. We step opposite it to go downhill.

Problem

Why Optimizers Exist

Real loss landscapes are noisy, ill-conditioned, and full of narrow valleys. Plain gradient descent can:

zig-zag in narrow valleys
stall when gradients vanish
diverge when gradients spike
need painful learning-rate tuning

Optimizers modify the update rule to stabilize steps and accelerate progress.

Diagram

One Diagram, One Intuition

The gradient points uphill. The optimizer chooses a direction and step size to move downhill.

Momentum adds “inertia” so updates keep going in consistent directions.

RMSProp and Adam adapt learning rates per-parameter using running statistics of gradient magnitudes.

Hands-On

Interactive Simulator

Compare how different optimizers move on the same loss surface. Start with Rosenbrock, then try Saddle or Waves. Watch for zig-zagging, overshoot, and stabilization.

Simulator

learning rate

0.020

beta1

0.900

beta2

0.999

speed

30.000

Tip: Rosenbrock has a narrow valley. Momentum and Adam usually track it better than plain GD.

What You’re Seeing

The grayscale background is the loss surface. Darker is lower loss. The white dot is the current parameter point (x, y). The black line is the trajectory created by the optimizer update rule.

Practical

When To Use What

GD: best for teaching; rarely used alone at scale.
Momentum: great default for smooth objectives; common in vision.
RMSProp: stable for non-stationary / noisy gradients; classic for RNNs.
Adam: strong “it just works” baseline; great for transformers and prototyping.

Tuning

Common Pitfalls

Learning rate too high: loss explodes (trajectory shoots out).
Learning rate too low: training crawls (tiny steps).
Adam with too-large lr can look stable but generalize worse.
Momentum too high can overshoot; too low behaves like GD.

Loading Lab...