Applied AI stage summary

Recurrent neural networks process a sequence one token at a time, carrying a hidden state forward. That design has two costs that no amount of hardware removes. Training cannot be parallelised along the sequence, because each step needs the previous step's hidden state, so a thousand-token sequence means a thousand sequential steps however many accelerators are available. And information from an early token has to survive every intermediate state to influence a later one, which degrades as sequences lengthen. Long short-term memory and gated recurrent units added gating that partly answered the second problem by making it easier to preserve or discard information, and did nothing at all about the first.

Self-attention answers both at once. For each position, three learned linear projections are computed: a query representing what this token is looking for, a key representing what this token offers to others, and a value carrying the information it contributes. The score between two tokens is the dot product of one query with the other key, divided by the square root of the key dimension so that scores do not grow with dimensionality and push the softmax into saturation where gradients vanish. The scores pass through a softmax to become weights, and the weights produce a weighted sum of the values. The whole operation is one matrix multiplication, so every position is computed at once rather than in turn.

Three additions make that mechanism into a working block. Because attention treats its input as a set, positional encoding adds position-dependent information to each embedding, so that a sentence and its shuffled reordering do not produce identical attention; the original design used fixed sinusoidal encodings, and rotary approaches encode relative position directly in the attention computation for better generalisation to lengths not seen in training. Multi-head attention runs several attention computations in parallel over separate subspaces and concatenates the results, so one head can learn a syntactic relation while another learns semantic similarity. And each sub-layer is wrapped in a residual connection and followed by layer normalisation, which is what lets very deep stacks train at all.

The architecture then splits three ways by what each token is permitted to attend to. Encoder-only models attend in both directions across the whole input and suit classification, entity recognition and similarity. Decoder-only models use causal masking so a token sees only what precedes it, which is what autoregressive generation requires. Encoder-decoder models keep both halves and suit translation and summarisation, where the full source has to be understood before the target is produced. The cost that is easy to forget is quadratic: every token attending to every other means the attention matrix grows with the square of sequence length, and that is the constraint behind sparse attention patterns, sliding windows and memory-aware implementations such as Flash Attention, which changes the memory access pattern rather than the algorithm.

A generative pre-trained transformer is the decoder half of the original architecture, trained on one objective: given every preceding token, predict a probability distribution over the vocabulary for the next one. Everything a base model appears to know is a side effect of getting good at that. It does not read characters or words. It reads tokens, which are sub-word units produced by byte-pair encoding: the algorithm starts from bytes and repeatedly merges the most frequent adjacent pair until the vocabulary reaches its target size. Common words become single tokens, uncommon ones are split into pieces, and nothing is ever out of vocabulary because any string decomposes into known pieces.

That has consequences people meet as billing surprises rather than as architecture. Rare words, technical terms and identifiers cost more tokens than their length suggests. Text in a language poorly represented in the vocabulary tokenises far less efficiently than English, which shrinks the effective context for exactly the users who can least afford it. And the context window, the maximum number of tokens the model can process in one forward pass, is not memory. Attention is not distributed evenly across positions, so material buried in the middle of a long context can receive less weight than material near the beginning or the end. Enlarging the window does not repair that, and it costs quadratically. Placing critical material at the ends, or retrieving only what is relevant, does.

Scaling is where sizing arguments go wrong. Early scaling work established that loss falls predictably as a power law in model size, data size and compute, and was read as an argument for spending compute on parameters. The compute-optimal correction is that model size and training tokens should be scaled together: for every doubling of model size, the number of training tokens should double too. The module works with a rule of thumb of roughly twenty training tokens per parameter, which is why a model of a hundred and seventy-five billion parameters trained on three hundred billion tokens is undertrained rather than merely large, and why smaller models trained on far more data have matched much larger predecessors at a fraction of the parameter count.

A base model is a text completer, not an assistant, and the difference is three stages. Pre-training consumes the overwhelming majority of the compute and produces language structure and world knowledge. Supervised fine-tuning trains on human-written instruction and response pairs and teaches the format and style of a helpful answer. Reinforcement learning from human feedback trains a reward model on pairwise human preferences, then optimises the language model against that reward while a divergence penalty keeps it near the fine-tuned checkpoint, which is what stops it exploiting the reward model. Direct preference optimisation skips the separate reward model, and feedback from a stronger model can stand in for human raters. Capabilities that appear only above a certain scale, such as learning a task from a few in-context examples, are real enough to plan around, though whether the sharp transitions are genuine or an artefact of exact-match scoring is still argued.

Zero-shot prompting gives the model an instruction and an input and nothing else, and it works for tasks the model has seen many instances of. What makes a zero-shot prompt effective is specificity about the output format, the role and the constraints, not politeness or length: a request for five measurable effects with a named region and an approximate figure for each produces something checkable, where a request to describe a topic produces an essay nobody can grade. System prompts sit in front of the user message and establish persistent behaviour, output format and safety boundaries, and they are the main control surface in a production system. Few-shot prompting supplies worked examples so the model can infer the pattern without any weight update; three examples usually establish a format, and example selection matters more than example count, because homogeneous or unbalanced examples bias the output toward the majority class.

Structured reasoning needs care. Chain-of-thought prompting is the research term for asking a model to produce intermediate steps before the final answer, and the published result is that visible intermediate steps improve accuracy on arithmetic, commonsense and symbolic tasks in sufficiently large models. The production guidance in this module is deliberately narrower: do not require disclosure of raw hidden reasoning by default, and instead ask for the artefact the reader can check. That means a concise rationale, cited evidence, stated assumptions, calculation steps, decision criteria or a verification checklist. Visible rationale is not proof of anything; a model can produce a fluent explanation that arrives at a wrong answer. Verification is what makes it useful: self-consistency across samples, independent tool checks, step-level validation, source-backed citation, and refusal when the evidence does not support an answer.

Retrieval-augmented generation is the structural fix for the fact that a model's knowledge is frozen at its training cutoff and it cannot reliably tell what it knows from what it has confabulated. The pipeline has three stages and each fails differently. Retrieve embeds the query and searches a vector store of pre-computed chunk embeddings for the closest matches, so retrieval quality depends on chunk size and overlap, on the embedding model's fit to the domain, and on whether keyword matching is combined with vector similarity. Augment inserts the retrieved chunks into the context alongside the query, with a system prompt that tells the model to answer from the provided context and to say when the context is insufficient. Generate produces the answer, and a well-built system attaches citations that tie specific claims to specific chunks.

The measurement follows the stages, which is the point of measuring at all. Retrieval precision asks whether the right chunks came back. Answer faithfulness asks whether the answer stays inside the retrieved context rather than adding unsupported claims. Citation accuracy asks whether each citation actually supports the claim it is attached to, and it can fail while the other two pass, because attribution is a generation-stage error. Retrieval does not eliminate fabrication. It moves the question from whether the model knows something to whether retrieval found it, which is a far better question because it can be instrumented, and it is why the retrieval half of a pipeline usually deserves more engineering attention than the generation half.

A convolutional network builds a hierarchy. Each convolutional layer slides a small learned filter across its input and computes a dot product at every position, producing a feature map that shows where that filter's pattern occurs. Pooling layers downsample, keeping the strongest activations and discarding spatial precision. Alternating the two produces early layers that respond to edges and textures, middle layers that respond to parts, and deep layers that respond to whole objects. Because those early and middle features are generic, transfer learning from a backbone pre-trained on a large image corpus is the normal starting point rather than training from scratch. Vision transformers take the other route, splitting an image into fixed-size patches, treating each patch as a token and applying the same self-attention machinery, which shows the architecture generalises beyond text when given enough data.

The tasks form a ladder of output granularity, and confusing two rungs is the most common category error in a vision brief. Classification returns one label for the whole image. Object detection returns a set of bounding boxes, each with a class and a confidence, and single-pass detectors made this fast enough for real-time use by reformulating detection as one regression over a grid rather than as propose-then-classify over many regions; non-maximum suppression then removes the duplicate boxes that overlap a stronger detection. Semantic segmentation labels every pixel, which is what applications need when shape matters, and it is made practical by upsampling paths with skip connections that restore the spatial detail pooling threw away. Instance segmentation separates individual objects of the same class, and panoptic segmentation does both at once. Evaluation follows: intersection over union measures how much a predicted box or mask overlaps the ground truth, and mean average precision averages precision across classes at one or more intersection-over-union thresholds, so a detector's headline score is meaningless without the threshold it was computed at.

Generative vision runs the process backwards. The forward process adds Gaussian noise to a training image over many timesteps until it is indistinguishable from noise. The model, conditioned on the timestep, is trained to predict the noise that was added, so at inference it can start from pure noise and iteratively remove its own predictions until an image emerges. Running that in a compressed latent space rather than in pixels is what made high-resolution generation affordable, and text conditioning enters through cross-attention on text embeddings. Guidance strength is a straight trade: push it up and the image tracks the prompt more closely while diversity falls and artefacts appear, pull it down and the prompt stops mattering.

The stage's cautionary case is about the seams rather than the models. In the Tempe fatality of March 2018, the perception system did detect the pedestrian seconds before impact, and then reclassified her repeatedly, each reclassification resetting the prediction of where she was going. Detection was not the failure. The absence of a track that persisted across reclassification was, compounded by a confidence threshold tuned to suppress false positives and by the absence of any conservative fallback for an ambiguous object. That is the shape of most vision failures in production: the classifier scores well on a benchmark and the pipeline around it has no answer for uncertainty. Bias belongs in the same category, because a model inherits the distribution of its training data and transfer learning carries that inheritance downstream into applications that never saw the original corpus.

Batch serving runs inference on a schedule and stores the results, so the prediction already exists when it is asked for. It suits work where the prediction can be computed before it is needed and where cost matters more than freshness. Real-time serving computes a prediction at request time, which is the only option when the decision has to be made before something else can happen, such as approving or declining a transaction while the customer waits. Most production systems are neither purely one nor the other: stable features computed in batch are combined at request time with volatile signals, which gives the cost profile of batch computation for the expensive parts and the responsiveness of real-time inference for the parts that change.

Around the serving choice sits a release loop with two manual gates, and each step exists because a specific failure has happened without it. A candidate is trained and evaluated under an experiment tracker. It is registered in a model registry, which stores the versioned artefact together with its lineage: the training data hash, the hyperparameters, the evaluation metrics, who trained it and when. Automated validation gates check metrics, bias and lineage before anything moves. Staging catches integration faults. A live experiment measures real-world impact. Promotion is a deliberate decision taken on evidence, and monitors are configured before traffic arrives rather than after. The registry is what makes rollback a lookup rather than a search for a checkpoint on somebody's laptop, and the promotion gates are the machine-learning equivalent of code review.

Two supporting pieces of infrastructure do more work than their names suggest. A feature store holds one definition of each engineered feature and serves it to both the training pipeline and the serving path, which is the only reliable defence against training-serving skew, where the same feature name means two subtly different computations and the model quietly degrades because the values are close enough to look correct. It also lets teams reuse each other's features instead of reimplementing them. A live experiment, splitting traffic between the current model and a candidate, catches what offline evaluation structurally cannot: feedback loops where the model's own outputs change the data it will next be judged on, novelty effects that inflate early results, contamination between the two groups, and guardrail metrics that move the wrong way while the headline metric improves.

Data drift, also called covariate shift, is a change in the distribution of the inputs. The model was fitted to data that looked one way and production data now looks another way, which is what happens to any classifier whose subject matter evolves, such as a spam filter meeting vocabulary and tactics that did not exist when it was trained. Concept drift is a change in the relationship between inputs and outputs. The features are still present and still measured the same way, but what they predict has moved. That is the harder case precisely because nothing about the inputs looks wrong, and it is the one that emptied a large iBuying programme: the property attributes were unchanged and their relationship to price was not, and the model kept buying at prices it had learned in a world that no longer existed.

The detection strategy follows directly from the distinction and needs both halves. Input monitoring compares production feature distributions against the training baseline using statistical distance measures, and it will catch covariate shift early because it needs no labels. Performance monitoring compares predictions against ground truth as labels arrive, and it is the only thing that catches concept drift, because by definition the inputs look normal. Neither alone is sufficient, which is why a drift dashboard is organised in lanes with an owner and a threshold for each, so that an alert names both the signal that moved and the person who acts on it. Thresholds turn a chart into a control: a distribution distance above a stated value, or a recall drop beyond a stated margin, should trigger investigation and a retraining decision rather than a discussion about whether the line looks unusual.

The failure that produces the largest losses is not a missing model, it is a missing circuit breaker. A monitored system that detects distribution shift and keeps acting at full volume has bought itself an alert and nothing else. The habit to take from this module is to pair every automated decision loop with a condition under which it pauses itself: a bounded exposure, a fallback to human review, or a suspension of automated action when precision drops below a stated floor. That condition is cheap to build before launch and almost impossible to retrofit during an incident.

Prompt injection is the entry that gets the attention and it deserves it, because it is structural rather than incidental. Direct injection is an attacker typing instructions into the same field the model reads, telling it to disregard its configuration. Indirect injection is more dangerous because the user never sees it: the attacker places instructions in content the system will later retrieve and process, such as a page it summarises, a message it reads or a record it queries, and the model follows them because it has no way to tell a retrieved document's content from its own instructions. The structural analogy is SQL injection, where user input and command travel in one string; here data and instructions travel in one context. That is why defences such as input filtering, output constraints, instruction hierarchy and human approval for consequential actions are mitigations layered in the expectation that some attacks succeed, rather than a fix.

Two attacks target the model's learning instead of its inputs. Data poisoning corrupts the training pipeline rather than the infrastructure, so the attacker needs access to a scraper, an annotation service or a public dataset rather than to the model. Backdoor poisoning is the variant that survives evaluation: a small number of examples pair a specific trigger with a target label, so the model behaves normally on clean inputs and produces the attacker's output whenever the trigger appears, and standard testing never sees it because the poisoned examples are a negligible share of the data. The defences are provenance tracking, anomaly detection over training data, analysis for clusters of suspiciously similar examples, and drawing on multiple independent sources so no single one can dominate. Adversarial examples work at inference instead: a perturbation computed from the gradient of the model's loss with respect to the input can flip a classification with high confidence while remaining imperceptible to a person, which is a property of how these models separate classes rather than a defect in one of them.

The last group attacks confidentiality and the supply chain. Model extraction rebuilds a proprietary model by querying its interface and training a surrogate on the responses, so returning fewer details, especially confidence scores, and rate limiting the interface are the practical defences. Membership inference asks whether a particular record was in the training set, which leaks information about the data rather than the model, and differential privacy during training is the defence with a mathematical guarantee behind it. Supply chain compromise is the one teams underestimate: a serialisation format that permits embedded code turns loading a downloaded model file into running untrusted code, so prefer formats that cannot execute, verify checksums, scan artefacts, pin versions, and extend the bill of materials to cover models and datasets as well as libraries. Ordered outward to inward, the layers are provenance, input filtering, architectural constraints, training-time hardening, rate limiting, runtime anomaly detection and an audit trail that makes an attack investigable whether or not it succeeded.

The EU AI Act sorts systems into four tiers by the risk they pose to health, safety and fundamental rights, and the obligations rise with the tier. Unacceptable risk covers practices that are prohibited outright, including manipulation that circumvents free will, exploitation of vulnerable groups, social scoring by public authorities, and real-time remote biometric identification in public spaces for law enforcement, subject to narrow exceptions. High risk covers systems used in areas such as critical infrastructure, education, employment and worker management, access to essential services including credit, law enforcement, migration and the administration of justice; these are not banned, but they carry risk management, data governance, technical documentation, record keeping, transparency, human oversight and accuracy requirements, and a conformity assessment before market. Limited risk covers systems that interact with people, generate synthetic content or infer emotion, and carries transparency duties: tell people they are dealing with an AI system and label synthetic content. Minimal risk carries no specific obligation.

The single most useful thing to carry out of this module is that the tier is a property of the deployment, not of the technology. The same face-matching model is minimal risk sorting a private photo library, high risk screening job applicants, and prohibited doing real-time identification in a public square. Nothing about the weights changed. That is why classification has to be redone for each deployment context rather than once per model, and why it has to be revisited when the context moves: new user populations, new territories, new data sources, or integration into a decision that now determines an outcome for a person can all change the answer.

Around the law sit the instruments that make compliance practicable. The NIST AI Risk Management Framework is voluntary and organises the work into four functions: Govern establishes policy and accountability, Map identifies risks and the stakeholders they fall on, Measure assesses and tracks them, and Manage prioritises and responds. Model cards standardise what a model's documentation should say: intended use, training data, evaluation conditions, performance disaggregated across groups, known limitations and ethical considerations, so that a downstream deployer can see the gaps rather than discover them. An impact assessment does the same job for a deployment, examining purpose, data provenance, disaggregated performance, the consequences of false positives and false negatives for the people affected, the route to human oversight and appeal, and the monitoring plan, and mature public-sector versions of the instrument produce a score that sets the level of scrutiny required. The United Kingdom has taken the other structural route, relying on sector regulators alongside a national institute that was recast from a safety remit to a security one, narrowing its focus to serious risks with security implications and to technical evaluation rather than sector regulation. The trade is legal certainty against adaptability.

The Act did not switch on all at once. Entry into force and application are separate events, and the duties phase in: prohibitions on the unacceptable-risk practices bind first, obligations on general-purpose models follow, general application and the transparency duties come later, and certain high-risk categories and systems embedded in regulated products come later still. The dates have also moved once already, which is the practical reason not to memorise them. Read the current schedule in the module, and plan against the earliest date that binds the specific thing you are building rather than against the headline date of the regulation. A compliance plan pinned to the wrong milestone is a plan to be late.

The second thing the phasing hides is who owes the duty. Obligations sit on providers, who build and place a system on the market, and separately on deployers, who put it to use in a particular context. Integrating somebody else's model into a hiring process, a credit decision or a triage workflow makes you a deployer of a high-risk system, with duties around transparency to the people affected, human oversight and incident handling, regardless of who trained the weights and regardless of whether you can see them. A provider cannot foresee every downstream use, which is exactly why the law splits the duty: the provider supplies documentation and declares intended uses, and the deployer is answerable for whether the system is appropriate where it has actually been put and whether the safeguards are genuinely in place.

The stage closes on a content moderation pipeline because it needs every part of the stage simultaneously. Content arrives and is normalised first, because normalisation is what defeats a whole class of evasion. It is then routed by modality to specialised models: language models for text, vision models for images, sampled frames for video, transcription for audio, and multimodal models for the cases where meaning lives in the combination and neither the text nor the image alone would be flagged. Raw scores are then enriched with context the classifiers cannot see, such as account history, how fast the content is spreading, region, and what it is replying to, because context is what separates reporting on an event from advocating one. A decision engine combines scores and context, and a human review layer takes the ambiguous cases, hears appeals, covers policy categories the models have not been trained on, and produces labelled data that feeds the next model.

The thresholds are the design, and none of them is a technical fact. Above a high confidence bound the system acts automatically; in a middle band it routes to a person with the model's assessment attached; below a floor it logs without acting. Move the bounds one way and more harmful material stays up, move them the other and more legitimate expression comes down. Class imbalance makes the usual headline metric worthless here, because a system that predicts no violation for everything is right almost all of the time and useful never; recall and precision are what decide whether the pipeline works, and they have to be reported per language and per category rather than in aggregate, since aggregate performance is dominated by the best-resourced language.

Two design constraints follow from taking the system seriously. Adversarial resilience has to be designed in, because adversaries adapt continuously: substituting visually similar characters, hiding text inside images, splitting a message across several posts, and switching languages mid-sentence to evade language-specific models. Normalisation and pre-processing handle whole classes of these, retraining handles specific new patterns, and the human review loop is the mechanism that turns each successful evasion into training data. Governance is the other constraint, and it is a design input rather than a documentation task afterwards: transparency about automated involvement, a working appeal route, independent audit for large platforms, and a model card that states which categories and which languages are covered, how performance varies across them, which failure modes are known, and where a person is in the loop.