Matrices in Machine Learning — Weight Matrices & Operations

Matrices in Machine Learning

Matrices are the fundamental data structure in machine learning. Nearly every operation — from data representation to model computation — involves matrix operations.

Data as Matrices

A dataset with m samples and n features is represented as an m x n matrix X:

X = | x11  x12  ...  x1n |   ← sample 1
    | x21  x22  ...  x2n |   ← sample 2
    | ...  ...  ...  ... |
    | xm1  xm2  ...  xmn |   ← sample m

Weight Matrices in Neural Networks

Each layer in a neural network has a weight matrix W and a bias vector b. The forward pass computes:

output = activation(W * input + b)

For a layer with 784 inputs (28x28 image flattened) and 128 neurons, W is a 128 x 784 matrix containing 100,352 learnable parameters.

Key Matrix Operations in ML

• Matrix multiplication: Forward and backward passes
• Transpose: Computing gradients (backpropagation)
• Inverse: Closed-form solutions for linear regression (normal equation)
• Eigendecomposition: PCA for dimensionality reduction
• SVD: Latent Semantic Analysis, recommendation systems

Batch Processing

Processing multiple samples at once is a matrix multiplication:

Batch output (32x128) = Batch input (32x784) * W^T (784x128)

This is why GPUs (optimized for matrix multiplication) dramatically accelerate ML training.

Gradient Computation

During backpropagation, gradients are computed using matrix operations:

dW = input^T * d_output    (gradient of weights)
d_input = d_output * W     (gradient to propagate back)

The entire training loop is essentially a sequence of matrix multiplications, additions, and element-wise operations.