LossFunctions

Lesson 8/11

Deep Learning • Objectives

Loss Functions

A loss function turns “how wrong the model is” into a single number to minimize. Your choice changes gradients, robustness to outliers, and how training behaves.

Idea

What A Loss Does

Training is an optimization problem: find parameters θ that minimize the expected loss.

Loss shapes the gradient. Gradient shapes the update. So loss isn’t just “measurement” – it’s the force that pushes learning.

θ ← θ − α · ∇θ L(θ)

Rule of Thumb

How To Pick

Regression: start with MSE, use Huber when outliers exist.
Binary classification: BCE (with logits).
Multi-class: Softmax cross-entropy.

The “best” loss is the one that matches your data noise model and your evaluation metric.

Diagram

Shapes Matter

MSE grows quadratically, so it heavily penalizes large errors (and outliers).

MAE grows linearly, making it robust but less smooth around zero.

Huber is the compromise: quadratic near zero, linear for large errors.

Interactive

Loss Curve Explorer

Simulation

Regression Outliers

Create outliers and see how the best-fit line changes depending on the loss. This is why robust losses exist.

Outlier Playground

outliers

magnitude

4.0

noise

0.25

Red points are outliers. MSE is very sensitive to them; MAE is robust but has a non-smooth kink; Huber trades off both.

Simulation

Classification Losses

Cross-entropy losses are built to turn probabilities into a training signal. Play with logits and see how loss and gradients react.

Binary Cross-Entropy

Treat logit as the model output before sigmoid.

logit

0.20

0.550

BCE

0.598

grad (∂L/∂logit)

-0.450

With BCE + sigmoid, the gradient simplifies to p − y.

Softmax Cross-Entropy

Adjust 3 logits and see the probability of the true class.

1.20

0.20

-0.50

0.645

0.237

0.118

0.439

Cross-entropy penalizes low probability on the true class. Softmax makes probabilities sum to 1.

Loading Lab...