Calculus

Calculus studies how quantities change. In AI/ML it shows up every time we optimize a model: we measure how a loss changes when we nudge parameters, then move parameters in the direction that improves the loss.

Core Ideas

• Limits: formalize “approaching” a value, enabling precise definitions of continuity and derivatives.
• Derivatives: local rate of change (slope). For a loss L(w), the derivative tells how sensitive the loss is to parameters w.
• Integrals: accumulation/area. In probability, continuous densities are normalized by integration and expectations are integrals.
• Multivariable calculus: gradients, Jacobians, Hessians — the language of deep learning.

Gradient (Multivariable Derivative)

For a scalar function L(w) where w ∈ R^d, the gradient is the vector of partial derivatives:

∇L(w) = [∂L/∂w₁, ∂L/∂w₂, …, ∂L/∂w_d]

In gradient descent we update parameters using a step size (learning rate) η:

w ← w - η ∇L(w)

Chain Rule (Why Backprop Works)

Neural networks are compositions of functions. The chain rule tells us how to compute derivatives through a composition efficiently:

If y = f(g(x)), then dy/dx = f'(g(x)) · g'(x).

Backpropagation is the chain rule applied repeatedly across layers, reusing intermediate results to avoid expensive recomputation.

What To Be Comfortable With

• Taking derivatives of common functions (polynomials, exp, log, sigmoid, tanh).
• Gradients in vector form; interpreting a gradient as the direction of steepest increase.
• Second derivatives / Hessians for curvature and optimization intuition.
• Basic integrals and expectations for continuous probability.