Statistics

Statistics turns data into conclusions. In ML, statistics helps you reason about estimation, generalization, uncertainty, and whether observed improvements are real or just noise.

Population vs Sample

You rarely see the full population; you observe a sample. Statistics studies how sample-based estimates behave.

Estimation

• Point estimates: mean, variance, regression coefficients.
• Bias–variance tradeoff: bias is systematic error; variance is sensitivity to sampling noise.
• Maximum likelihood (MLE): choose parameters maximizing p(data | θ).
• MAP: adds a prior, maximizing p(θ|data).

Hypothesis Testing & Confidence

When comparing models (A/B tests, new architecture vs baseline), you want to know if gains are statistically significant and practically meaningful.

• p-value: how surprising your result is under a null hypothesis.
• Confidence interval: plausible range for an estimated quantity.
• Effect size: how large the improvement is (not just whether it exists).

Generalization & Overfitting

• Train/validation/test splits approximate performance on unseen data.
• Cross-validation reduces variance for small datasets.
• Regularization and early stopping reduce overfitting.

Useful Mental Models

• Correlation ≠ causation; confounding can create misleading relationships.
• Multiple comparisons can produce false “wins” if you test many variants.
• Data leakage can inflate metrics; guard the test set carefully.