Stage 4 summary. Data and AI

A model is a function fitted to examples, so whatever pattern is in the examples is what it learns, including the sampling bias, the field left blank after a migration, and the duplicates that make one customer look like four. A larger model does not remove those patterns; it reproduces them more fluently. There is a structural reason too. A model is retrained rarely and behaves the same way in between, while the data underneath it changes continuously, so most of the variation in a system's behaviour over a year comes from the foundation.

An embedding places a piece of text at a point in a space arranged so that related material lands close together. Nothing about the words is preserved; the neighbourhood is. That is why a search for staff turnover returns a document that says attrition, and why a search for a part number can miss, because a rare code carries little of the contextual signal that positioned it. Three consequences bite delivery teams: a real space has hundreds of dimensions rather than two, vectors from one model cannot be compared with another's, and changing the embedding model means re-embedding the whole corpus.

AI-ready data is a readiness test applied to a specific dataset for a specific intended use, not a file format or a badge. The UK guidance published on 19 January 2026 sets four components a dataset addresses: technical optimisation, quality and adherence to standards, legal and regulatory compliance, and responsible management. The one most often lost is organisational: whether anyone is funded and accountable to keep the dataset in that state next quarter. A feature store then holds one definition for both training and live prediction, which is what stops training-serving skew, the gap that opens when the same feature is calculated twice by different people months apart.

Retrieval-augmented generation has the model read rather than recall: a retriever searches an index for passages relevant to the question, those passages go in front of the model, and the answer is written from them. The consequence for a data team is accountability: an answer from training alone cannot be traced or notice that the policy changed last week, while an answer from retrieval can point at the passage it used. The six stages each hand on a different object: chunk produces passages, embed produces vectors, index produces the searchable artefact with its refresh schedule and backup story, retrieve produces candidates, rerank produces ranked context, and generate produces the only artefact a reader sees.

Because only the last is visible, an unmeasured pipeline gives nobody a way to tell a retrieval problem from a generation problem. Logging which passages were retrieved and which were shown is the retrieval equivalent of lineage, and evaluation loops back to the chunking decision rather than sitting at the end. Chunking constrains everything after it: small passages match sharply but cut answers in half and strip the qualifying sentence, while large passages hold an argument together but blur their position so a specific query fails to surface them. The trade cannot be won, only chosen against the questions the system will really be asked.

The vocabulary of vector databases implies a separate product more often than the architecture requires one, and the choice is better made on governance than on benchmarks. Keeping vectors beside the relational data keeps retrieval inside the access controls, backups and lineage already run. Approximate search is a quality setting rather than only a speed setting, so recall has to be measured against exact results, and hybrid retrieval combining keyword and vector search is what recovers the part numbers and surnames a demonstration audience reaches for first. The index is also a governed copy: it goes stale, it is read by people with different entitlements, and it belongs in deletion work.

Curating a training corpus is not tidying before training. Deduplication removes exact and near-duplicate documents, which otherwise waste compute, raise the chance of verbatim reproduction and contaminate evaluation by putting benchmark questions in the training set. Filtering decides what is admitted, and every filter costs on both sides: too little and the model learns the sludge, too much and the rare topics and technical registers that made the corpus worth having go with it. Licensing establishes entitlement, and it is where many corpora fail an audit because the crawl came first and the rights question second.

When Shumailov and colleagues published the model collapse result in Nature in July 2024, showing that models trained repeatedly on model-generated content drift from the real distribution and lose the rare cases first, the lesson was about data rather than models. A corpus scraped from the open web is no longer a sample of what people wrote; it is that mixed with machine output in an unknown and rising proportion. If you cannot say where a document came from, you cannot say whether it is evidence or an echo. Synthetic data has three defensible uses, in test environments, in balancing genuinely rare classes and as a stand-in for sharing, and two failure modes: it cannot add information that was never there, and it is not automatically anonymous.

Content Credentials, published under the C2PA specification, carry that history for media as a chain of signed claims rather than a single badge, so verification walks the chain back to each signer. The reading that keeps a team out of trouble is that a resolved chain gives history you can show, while an absent credential establishes nothing: most files carry none and many pipelines strip metadata routinely. In the European Union the same discipline is written down in two places aimed at two actors: Article 10 of Regulation (EU) 2024/1689 governs the data used to build high-risk systems, and Article 53 puts copyright policy and a public summary of training content on providers of general-purpose models.

A business glossary is about language: the agreed list of terms, each with one written definition, so that active customer means the same thing in a report, a contract and a model. Its unit is a term. A data catalogue is about assets: an inventory of what each dataset is, who owns it and how far it can be trusted. Its unit is a dataset. Merging the two usually satisfies neither audience, and both share a weakness: a description typed by hand decays the moment the pipeline behind it changes, and nothing in the tool notices.

Active metadata answers that decay. Declared metadata is asserted by a person and is true on the day it is written; active metadata is emitted by the running system, so the fact that a table has not been written for eleven days against a daily schedule arrives on its own. Lineage is the same argument at larger scale. Drawn on a whiteboard it is documentation and rots like documentation; captured at run time through a standard such as OpenLineage it is evidence, because the execution that moved the data is what emitted the record.

Column-level lineage is what makes that record decision-grade. Table-level lineage says a report reads a table; column-level lineage says whether the column you are about to change is one the report actually uses, and it separates a direct dependency, where the output is derived from the input, from an indirect one, where a join or filter shaped the result without the column appearing in it. An impact analysis then produces a short list with named owners rather than a change freeze. The adoption problem deserves naming honestly: a catalogue competes not with a rival catalogue but with a colleague who answers confidently in ninety seconds, and a design that loses on speed and trust will be bypassed.

Two people arrive at a meeting with the same chart title and different numbers, and both are right, because the charts were never counting the same thing. Trace each back and the calculation splits into three decisions: which source table, which filter says what counts, which time window. The board pack counts customers who paid, from the billing tables, over a rolling quarter; the operations screen counts anyone who signed in, from the product event stream, over the same quarter. One decision differs and that is enough. Drift is expensive out of proportion to its cause: the reconciliation recurs every cycle, and a leadership team shown two contradictory figures discounts every figure afterwards.

A semantic layer is where business measures are defined once, between the warehouse and everything that consumes numbers from it. It typically holds four things: entities and their join paths, the dimensions a measure may legitimately be sliced by, the measures and metrics with their filters and time handling, and the access rules, so who may see which rows is decided where the rows get their meaning. The Open Semantic Interchange initiative, launched in September 2025, is the industry effort to make those definitions portable, standardising how semantic metadata is exchanged so a definition is written once rather than re-encoded per tool. It standardises the exchange format, not how an engine computes a result, and is best treated as an unfinished promise to test against your own suppliers.

Agent access turns this from tidiness into urgency. Every previous consumer of a metric was a person, and someone who knows the business stops when a customer count jumps ten per cent in a week. That instinct was unpaid quality control the whole reporting estate relied on. An assistant reaching a warehouse through a standard connector can enumerate tables, write correct SQL against whichever of four plausible definitions it picked, and answer with no hedging. The defence is a definition written down properly, carrying source, grain, filters, time window and a named owner, where both the dashboard and the agent have to ask for it.