Optimization

Optimization is the process of choosing parameters that minimize (or maximize) an objective. Training an ML model is optimization: we pick weights that minimize a loss on data while generalizing well to unseen examples.

Problem Setup

Most supervised learning problems look like:

min_w  L(w) = (1/n) Σᵢ ℓ(f(xᵢ; w), yᵢ) + λ R(w)

• ℓ is the per-example loss (MSE, cross-entropy, etc.).
• R(w) is a regularizer (e.g. L2 weight decay) that discourages overfitting.

First-Order Methods

First-order methods use gradients. They scale to huge models because gradients can be computed efficiently with backpropagation.

• Gradient Descent: full dataset per step (expensive for large data).
• SGD / Mini-batch SGD: estimate gradients from a batch; adds noise that can help escape shallow minima.
• Momentum: smooths updates by accumulating a velocity.
• Adaptive methods: Adam/RMSProp tune step sizes per-parameter.

Second-Order Intuition

Curvature matters. The Hessian (matrix of second derivatives) describes how the loss bends; steep curvature can require smaller steps for stability.

H(w) = ∇²L(w)

Full second-order methods are often too expensive for deep nets, but curvature ideas still guide learning rate schedules, normalization, and optimizer design.

Constraints & Regularization

• Constraints (e.g. non-negativity) can be handled via projections or parameterizations.
• Regularization trades off fit vs simplicity; common examples include weight decay, dropout, and early stopping.

Practical Checklist

• Normalize inputs; watch loss curves for divergence or plateaus.
• Start with Adam, then tune learning rate (often the most important hyperparameter).
• Use learning-rate schedules; consider warmup for large models.
• Track validation metrics to detect overfitting early.