Think of a light switch — either on or off. The output is completely binary, with an abrupt discontinuity at zero.

The derivative is zero everywhere except at x=0 where it is undefined. This makes gradient-based learning impossible — the network cannot determine how to adjust weights.

Binary classifiers in simple perceptrons. Largely replaced by sigmoid in modern practice. Sometimes used in binary neural networks for edge deployment.

• ✓ Extremely simple to compute
• ✓ Clear binary decision boundary
• ✓ Historically important in early perceptrons

• ✗ Not differentiable at zero
• ✗ Cannot use gradient descent directly
• ✗ Dead gradient problem everywhere
• ✗ No probabilistic interpretation

A straight line through the origin. No matter how many layers use linear activations, the entire network collapses to a single linear transformation — depth provides no representational benefit.

The gradient is constant (α) everywhere — no vanishing or exploding gradient caused by this function itself. However, stacking linear layers collapses to one linear layer, so depth is useless.

The output layer for regression tasks where the target is continuous and unbounded. Never used in hidden layers of a network meant to learn non-linear functions.

• ✓ No vanishing gradient
• ✓ Simple and fast
• ✓ Useful for regression output layers

• ✗ Network depth is wasted — all layers collapse to one
• ✗ Cannot learn non-linear patterns
• ✗ Cannot produce probabilities

An S-shaped curve that is steep near zero and flat at the extremes. Large positive inputs saturate toward 1, large negative inputs toward 0. The output is always a valid probability.

f'(x) = f(x)(1 - f(x)), maximized at 0.25 when x=0. At saturation (|x| > 4), the gradient approaches zero — vanishing gradient in deep networks.

The output layer for binary classification. In LSTMs, used for gate activations. Avoid in deep hidden layers — use ReLU family instead.

• ✓ Outputs interpretable as probabilities
• ✓ Smooth, differentiable everywhere
• ✓ Clear probabilistic meaning
• ✓ Bounded output prevents explosion

• ✗ Vanishing gradient at saturation
• ✗ Output not zero-centered (shifts gradients positive)
• ✗ Expensive to compute (exp)
• ✗ Not suitable for deep hidden layers

A steeper, zero-centered version of sigmoid. Outputs negative for negative inputs, positive for positive — exactly what we want for weight updates to go in either direction symmetrically.

f'(x) = 1 - tanh²(x), peaking at 1.0 when x=0. Still suffers from vanishing gradients at saturation, but less severely than sigmoid.

RNN hidden states, LSTM cell gates, older feedforward networks. Still preferred over sigmoid in hidden layers due to zero-centering.

• ✓ Zero-centered output (symmetric gradients)
• ✓ Stronger gradient than sigmoid near origin
• ✓ Bounds output to avoid explosion
• ✓ Good for RNNs and LSTMs

• ✗ Vanishing gradient at saturation
• ✗ Computationally expensive (two exp calls)
• ✗ Still saturates for large |x|

A ramp function. When a neuron sees a negative input, it outputs zero — it becomes inactive. When positive, it passes the signal unchanged. This sparsity is computationally efficient and biologically plausible.

Gradient is 1 for positive inputs, 0 for negative. No vanishing gradient on the positive side. The dying ReLU problem: neurons with consistently negative preactivations output zero gradient — permanently dead.

Default choice for hidden layers in CNNs, MLPs, and most feedforward architectures. Combined with batch normalization for deep networks.

• ✓ No vanishing gradient (positive side)
• ✓ Computationally trivial (max operation)
• ✓ Induces sparse activations (~50% zeros)
• ✓ Accelerates convergence significantly
• ✓ Works well with batch normalization

• ✗ Dying ReLU: neurons can become permanently inactive
• ✗ Not zero-centered output
• ✗ Not differentiable at 0
• ✗ Unbounded output can cause instability

A tiny leak for negative values — just enough gradient to rescue dying neurons. Visually almost identical to ReLU, but with a shallow slope instead of zero on the left.

1 for x>0, α for x≤0. No dead neuron problem. Small but consistent gradient for all inputs ensures every neuron can recover.

Anywhere you would use ReLU but fear dead neurons — particularly useful with very deep networks or aggressive learning rates.

• ✓ No dying neuron problem
• ✓ Nearly as fast as ReLU
• ✓ Allows gradients to flow for negative inputs
• ✓ Simple drop-in replacement for ReLU

• ✗ α is a hyperparameter requiring tuning
• ✗ Not zero-centered
• ✗ Inconsistent performance advantage over ReLU
• ✗ Not always better than ReLU in practice

The network itself decides how much to leak for each neuron. Some neurons may learn high α (nearly linear), others near-zero α (nearly ReLU). Each neuron adapts to its role.

Backprop through x and through α. For negative x, dL/dα = dL/df * x, allowing α to be updated. Introduces extra parameters but minimal overhead.

Used in ResNet and other image classification networks. Introduced by He et al. (2015), showing improvement over ReLU for deep residual networks.

• ✓ Learnable negative slope adapts to data
• ✓ Can outperform fixed Leaky ReLU
• ✓ Minimal parameter overhead
• ✓ Improved performance in image tasks

• ✗ Adds parameters to optimize
• ✗ Can overfit on small datasets
• ✗ Initialization matters
• ✗ More complex to implement than ReLU/Leaky ReLU

Imagine Leaky ReLU but with a smooth curve instead of a line for negatives. The exponential form means the negative output saturates at −α, pushing mean activations closer to zero.

1 for x>0, α·eˣ for x≤0. Smooth everywhere, no kink at origin. Negative saturation region still has vanishing gradient for large |x|.

Deep fully-connected networks, residual networks, situations where smooth gradients matter. Introduced as an improvement over ReLU with better statistical properties.

• ✓ Smooth everywhere (differentiable at 0)
• ✓ Negative outputs push mean toward zero
• ✓ Reduces bias shift during training
• ✓ No dying neuron problem

• ✗ Computationally more expensive than ReLU (exp)
• ✗ Saturates for large negative inputs
• ✗ α requires tuning
• ✗ SELU generally preferred when self-normalization is needed

SELU is the activation equivalent of batch normalization baked into the function itself. With the right weight initialization, activations automatically stay normalized — no batch norm needed.

λ for x>0, λ·α·eˣ for x≤0. The specific constants ensure the network is a contraction mapping — activations converge to normalized distributions.

Self-normalizing Neural Networks (SNNs). Works best in deep feedforward architectures with no residual connections, used with AlphaDropout.

• ✓ Self-normalizing without batch normalization
• ✓ Enables very deep feedforward networks
• ✓ Theoretically grounded (Klambauer et al. 2017)
• ✓ Consistent performance across depths

• ✗ Requires specific weight initialization (LeCun normal)
• ✗ Requires specific architecture (no skip connections, dropout)
• ✗ Batch norm often used anyway, reducing advantage
• ✗ Less effective in CNNs

Imagine smoothing out the kink in ReLU. At x=0, ReLU has a sharp corner — Softplus rounds this corner. For large |x|, softplus ≈ ReLU for positive and ≈ 0 for negative.

f'(x) = 1/(1+e⁻ˣ) = sigmoid(x). Derivative is sigmoid! This means gradient vanishes slightly for large negative x but never reaches exactly zero.

Variational Autoencoders (VAE) where smooth activations help optimization. Also used when differentiability everywhere is required (e.g., second-order optimization).

• ✓ Smooth and differentiable everywhere
• ✓ No dead neurons
• ✓ Approximates ReLU with smooth gradient
• ✓ Useful in variational models (VAEs)

• ✗ Computationally expensive (log + exp)
• ✗ Not sparse like ReLU
• ✗ Slightly inferior to ReLU in practice
• ✗ Output not zero-centered

Like tanh but using |x| in the denominator rather than an exponential. This makes it cheaper to compute and slower to saturate — inputs must be larger to push the function into its flat regions.

f'(x) = 1/(1+|x|)². Decays as 1/x² rather than exponentially, so gradients vanish more slowly than tanh.

Situations where tanh is appropriate but computational budget is limited. Found in some older architectures and specialized applications.

• ✓ Cheaper to compute than tanh (no exp)
• ✓ Slower saturation preserves gradients longer
• ✓ Zero-centered output
• ✓ Good alternative in resource-constrained settings

• ✗ Less commonly used than tanh or ReLU
• ✗ Less framework support
• ✗ Not as well studied as alternatives
• ✗ Polynomial decay still causes some gradient vanishing

GELU asks: how likely is this input to be active under a Gaussian? Large positive inputs are almost certainly active; large negatives almost certainly inactive; near zero, there's uncertainty — GELU applies a smooth, probabilistic gate.

Complex but smooth. Has a slightly negative region near x ≈ -0.17, then rises steeply. This non-monotone property correlates with improved optimization landscapes.

Default activation in transformers (BERT, GPT-2/3/4, RoBERTa, T5). Also used in vision transformers (ViT) and other modern architectures.

• ✓ State-of-the-art performance in NLP
• ✓ Smooth, non-monotone shape aids optimization
• ✓ No dead neuron problem
• ✓ Stochastic interpretation aids regularization

• ✗ More expensive to compute (requires tanh approximation)
• ✗ Complex derivative
• ✗ Less interpretable than ReLU
• ✗ May not improve over ReLU in non-NLP tasks

The input gates itself — large positives pass through near-unchanged (sigmoid→1), near-zero inputs are suppressed, and the function has a slight negative region that acts as implicit regularization.

f'(x) = f(x) + σ(βx)(1-f(x)). Smooth everywhere. The slight non-monotonicity (the dip below zero) is believed to help escape poor local optima.

Used in EfficientNet and other modern vision models. Good default for deep CNNs and MLPs. β=1 (called SiLU in PyTorch) is the standard choice.

• ✓ Consistently outperforms ReLU on deep networks
• ✓ Smooth and differentiable
• ✓ Non-monotone aids optimization
• ✓ β=1 works well without tuning

• ✗ More expensive than ReLU (sigmoid computation)
• ✗ β introduces another hyperparameter
• ✗ Marginal improvement over GELU
• ✗ Not as widely adopted as GELU in NLP

Mish is like Swish but with a richer negative region — the curve dips more deeply before rising. This creates a self-regularizing effect and strong gradient flow near the origin.

Complex composite derivative involving softplus, tanh, and their derivatives. Always non-zero, smooth everywhere, with stronger gradient near origin than Swish.

YOLOv4 and derivatives, some vision transformers. Strong choice for object detection. Generally competitive with Swish across domains.

• ✓ Strong performance on object detection
• ✓ Non-monotone with smooth negative region
• ✓ Self-regularizing behavior
• ✓ Continuous and differentiable everywhere

• ✗ Computationally expensive (tanh + softplus + log)
• ✗ Complex implementation
• ✗ Marginal improvement over Swish on most tasks
• ✗ Less widely supported in frameworks

Imagine each class competing for probability mass. Softmax amplifies differences — the largest value dominates exponentially, squashing smaller ones. The outputs always sum to exactly 1.

Jacobian (not scalar derivative). For i≠j: ∂fᵢ/∂xⱼ = -fᵢfⱼ. For i=j: ∂fᵢ/∂xⱼ = fᵢ(1-fᵢ). Combined with cross-entropy loss, simplifies to (predicted - true).

Output layer for multi-class classification in virtually every classifier. Also used in attention mechanisms (scaled dot-product attention in transformers).

• ✓ Outputs valid probability distribution (sum=1)
• ✓ Standard for multi-class classification
• ✓ Numerically stable with log-softmax
• ✓ Cross-entropy gradient is very clean

• ✗ Operates on vectors not scalars
• ✗ Sensitive to temperature — can become too peaked
• ✗ Not suitable for hidden layers
• ✗ Computationally linked across all outputs