Mathematical foundations for data

A vector is an ordered list of numbers: [3.2, 0.8, 1.5]. In data contexts, vectors represent things. A product in an e-commerce catalogue might be represented as a vector of feature values: [price, category_encoded, average_rating, days_since_launch]. A word in a language model is represented as a 768-dimensional or 1536-dimensional embedding vector where proximity in that high-dimensional space corresponds to semantic similarity.

A matrix is a rectangular array of numbers. A dataset with 10,000 rows and 15 features is a 10,000 by 15 matrix. Multiplying that matrix by a weight matrix transforms the features into a new representation. This operation, matrix multiplication, is the core computation of every neural network layer, every linear regression, and every principal component analysis.

The dot product of two vectors measures their similarity: it is the sum of element-wise products. If two user preference vectors point in the same direction in feature space, their dot product is large. If they are orthogonal (no shared preference), it is zero. This is why collaborative filtering works: users with similar preference vectors get similar recommendations.

Transposing a matrix (flipping rows and columns) is a constant operation in data processing: converting column vectors to row vectors, reshaping batches for neural network layers, and computing covariance matrices all use transposition. The covariance matrix of a dataset is computed as the product of the data matrix transposed with itself, divided by the number of observations minus one.

Training a machine learning model means finding the parameter values that minimise a loss function: a number that measures how wrong the model's predictions are. Gradient descent is the algorithm that does this. The gradient of the loss with respect to each parameter tells you the direction of steepest increase. To minimise the loss, you move in the opposite direction, scaled by a learning rate.

Each update step is: parameter = parameter minus (learning rate times gradient). Repeat this millions of times across many examples and the model converges on parameters that produce low loss. The learning rate is the most consequential hyperparameter: too large and the model oscillates; too small and training takes impractically long or stalls in a local minimum.

Stochastic gradient descent (SGD) computes the gradient on a small random batch of examples rather than the full dataset. This introduces noise into the gradient estimate but reduces computation per step enormously. Mini-batch SGD with batch sizes of 32 to 256 is the standard in practice. Adaptive optimisers like Adam adjust the learning rate per parameter based on recent gradient history, which typically accelerates convergence.

Backpropagation is the algorithm that computes gradients efficiently in neural networks. It applies the chain rule of calculus to propagate error signals from the output layer backward through each layer, computing each layer's contribution to the loss. Without backpropagation, training deep networks would be computationally infeasible.

Bayes' theorem states that the probability of a hypothesis given observed evidence equals the prior probability of the hypothesis multiplied by the likelihood of the evidence given the hypothesis, divided by the probability of the evidence. Written compactly: P(H|E) = P(E|H) x P(H) / P(E).

The power of Bayes is in updating beliefs as new evidence arrives. A spam filter has a prior belief about the probability of an email being spam. When it observes specific words (the evidence), it updates that belief. After seeing many words, the posterior probability is high enough to classify the email. This is Naive Bayes classification: it assumes features are conditionally independent given the class label (a simplification, but one that works well in practice for text).

Medical testing is the canonical illustration of Bayes. A test with 99% sensitivity and 99% specificity for a disease affecting 0.1% of the population will produce more false positives than true positives when applied to the general population. Specifically, of 1,000 people tested: 1 has the disease and tests positive (true positive), 0 has the disease and tests negative (false negative in 1% of cases), and approximately 10 healthy people also test positive (false positive rate of 1% applied to 999 healthy people). The positive predictive value is 1/11, or about 9%. Understanding base rates is essential for interpreting model outputs.

Shannon entropy quantifies the average uncertainty in a probability distribution. A fair coin has maximum entropy (1 bit) because both outcomes are equally likely. A biased coin that lands heads 99% of the time has very low entropy because the outcome is almost entirely predictable. Entropy is the foundation of information theory, compression algorithms, decision trees, and the cross-entropy loss function used to train classification models.

Cross-entropy loss measures the difference between a model's predicted probability distribution and the true distribution (usually a one-hot vector where the correct class has probability 1 and all others have probability 0). Minimising cross-entropy during training pushes the model to assign high probability to correct predictions.

Mutual information measures how much knowing one variable reduces uncertainty about another. It is used in feature selection (high mutual information between a feature and the target means the feature is informative), in clustering evaluation, and in the information bottleneck method for training compressed representations in neural networks.

KL divergence (Kullback-Leibler divergence) measures how much one probability distribution differs from a reference distribution. It appears in variational autoencoders, in reinforcement learning policy optimisation, and in evaluating how well a fitted statistical distribution matches empirical data.