Scaling and Cost

Every model parameter occupies memory. In full precision (FP32), each parameter requires 4 bytes. A 7-billion-parameter model therefore requires approximately 28 GB just for its weights, before accounting for activations, gradients, and optimizer state during training. An NVIDIA A100 GPU has 80 GB of high-bandwidth memory (HBM). A consumer RTX 4090 has 24 GB. The arithmetic determines what you can run and where.

During training, memory consumption is roughly 4x the model size because you store the weights, gradients, optimizer states (which for Adam are two additional copies), and activations. A 7B parameter model at FP32 requires approximately 112 GB for training, which exceeds a single A100. This is why distributed training across multiple GPUs is standard for large models.

During inference, memory requirements drop dramatically because you only need the weights and a small amount of activation memory. A 7B model at FP16 (2 bytes per parameter) requires approximately 14 GB, which fits on a single consumer GPU. This asymmetry between training and inference memory is the foundation of the cost optimisation strategies that follow.

Quantisation reduces the numerical precision of model weights from 32-bit floating point (FP32) to lower bit widths: FP16 (half precision), INT8 (8-bit integer), or INT4 (4-bit integer). Each reduction halves or quarters the memory footprint and increases throughput because lower-precision arithmetic is faster on modern GPU hardware.

Post-training quantisation (PTQ) applies quantisation to an already-trained model without retraining. The simplest approach rounds each weight to the nearest representable value at the target precision. More sophisticated methods like GPTQ and AWQ analyse the model's weight distributions to minimise the accuracy loss from rounding.

Quantisation-aware training (QAT) simulates quantisation during training, allowing the model to adapt its weights to the reduced precision. QAT typically preserves more accuracy than PTQ but requires retraining.

The practical impact is dramatic. Llama 2 70B at FP16 requires two A100 GPUs (140 GB). At INT4 with GPTQ, the same model fits on a single A100 (roughly 35 GB) with minimal accuracy loss on most benchmarks. For many production use cases, INT8 quantisation is the sweet spot: negligible accuracy loss with a 2x memory reduction and significant throughput improvement.

Knowledge distillation trains a smaller "student" model to mimic the behaviour of a larger "teacher" model. Instead of training the student on the original labels alone, you train it on the teacher's output probabilities (soft labels). These soft labels carry richer information than hard labels: they encode the teacher's uncertainty and the relationships between classes.

The technique was formalised by Hinton, Vinyals, and Dean in 2015. The core insight is that the dark knowledge in the teacher's probability distribution, which classes the teacher considers similar, what it finds ambiguous, carries information that hard labels discard. A student trained on soft labels often outperforms an identically sized model trained on hard labels alone.

In the LLM era, distillation has become the primary method for creating smaller, deployable models. OpenAI's GPT-4 is widely believed to have been distilled into smaller models for specific tasks. The Alpaca and Vicuna projects demonstrated that fine-tuning a small open model on outputs generated by a larger model could produce surprisingly capable results. This approach, sometimes called "distillation through imitation," is now standard practice.

Pruning removes weights, neurons, or entire layers from a trained model that contribute little to its output. The lottery ticket hypothesis (Frankle and Carlin, 2019) showed that large neural networks contain sparse subnetworks that, when trained in isolation, match the full network's performance. This suggests that most parameters in a trained model are redundant.

Unstructured pruning sets individual weights to zero based on their magnitude. It can achieve high sparsity (90%+ zeros) but requires specialised hardware or software to exploit the sparsity for actual speedup. Standard GPU matrix multiplication does not benefit from scattered zeros.

Structured pruning removes entire neurons, attention heads, or layers. It produces models that are genuinely smaller and faster on standard hardware because the pruned dimensions are physically removed. The trade-off is that structured pruning is more aggressive and typically requires fine-tuning after pruning to recover accuracy.

In practice, quantisation and distillation are more widely used than pruning for LLMs. Pruning shines for convolutional neural networks and smaller models where structured removal of filters or channels produces immediate speedup on edge devices.

Edge deployment runs inference on the end-user's device (phone, browser, IoT sensor) rather than in a data centre. The benefits are compelling: zero network latency, no per-request API cost, and data never leaves the device (important for privacy). The constraint is compute: a mobile phone has far less memory and processing power than a cloud GPU.

Frameworks like TensorFlow Lite, ONNX Runtime, and Core ML (Apple) optimise models for edge hardware. Techniques from this module, quantisation to INT8, pruning, distillation, are the prerequisite for edge deployment. A 7B parameter model will not run on a phone; a 1B distilled, INT4-quantised variant might.

The decision between cloud and edge is not binary. Many production systems use a hybrid approach: a small, fast model runs on the device for common cases, and complex queries are forwarded to a larger cloud model. This pattern, called speculative decoding in the LLM context, reduces average latency and cost while maintaining quality for difficult inputs.

Training a large model is expensive, but it happens once (or on a schedule). Inference happens for every user request, every second of every day. For most production ML systems, inference cost dominates the total cost of ownership. OpenAI reportedly spends more on inference infrastructure than on training.

The key levers for reducing inference cost are: batching(processing multiple requests simultaneously to maximise GPU utilisation), caching (storing and reusing predictions for repeated inputs), model selection (routing easy queries to small models and hard queries to large ones), and the techniques from this module (quantisation, distillation, pruning).

A practical framework for inference cost estimation: cost per request equals (GPU cost per second) times (latency per request) divided by (batch size). An A100 at $2/hour serving a 7B INT8 model at 50ms per request with a batch size of 32 costs approximately $0.0001 per request, or $0.10 per thousand requests. The same model at FP16 with batch size 1 costs roughly $0.003 per request, a 30x difference. Every optimisation technique in this module directly reduces this number.