Linear Algebra

Linear algebra is the math of vectors, matrices, and linear transformations. Modern ML is implemented as large sequences of matrix multiplications plus nonlinearities — so linear algebra is the backbone of efficient model computation.

Vectors, Matrices, Tensors

• Vector: an ordered list of numbers (features, embeddings, gradients).
• Matrix: a 2D table (linear layers, attention projections, covariance).
• Tensor: generalization to N dimensions (batch × sequence × features).

Matrix Multiplication Intuition

A matrix W defines a linear map. For input vector x, the output y = Wx is a new vector whose components are weighted sums of x.

y = Wx + b   (a typical neural network layer)

Dot Product, Norms, Angles

The dot product measures alignment and is fundamental to similarity search and attention.

x · y = Σᵢ xᵢ yᵢ

• Norms (e.g. ||x||₂) measure vector length and are used in regularization and normalization.
• Cosine similarity compares direction (angle) rather than magnitude.

Eigenvalues, Eigenvectors, SVD

Decompositions reveal structure inside matrices — useful for dimensionality reduction and understanding geometry.

• Eigen: Av = λv captures directions v that a matrix scales by λ.
• SVD: A = U Σ Vᵀ generalizes eigen ideas to any matrix; PCA is closely related to SVD on centered data.

What To Be Comfortable With

• Shapes: verifying dimensions in matrix products.
• Transpose, inverse (when it exists), rank, and conditioning.
• Orthogonality and projections (least squares, embeddings).
• Gradients of simple matrix expressions (useful for ML derivations).