Data for AI foundations

A model is a function fitted to examples. Whatever pattern is present in those examples is what the model learns, including the patterns nobody intended it to learn: the sampling bias in how cases were recorded, the field that was left blank for six months after a system migration, the duplicate customer records that make one person look like four. Swapping a larger model into that setting does not remove the pattern. It reproduces the pattern more fluently.

So the work that changes a disappointing system is usually work on the examples: fixing labels, removing duplicates, filling documented gaps, and narrowing the task until the available data can support it. That order of operations is the argument of Andrew Ng’s Machine Learning Yearning, which sets out how to diagnose the errors in a machine learning system and prioritise the most promising direction for a project rather than iterate on the model and hope.

The definition of quality that survives contact with AI work is not a score out of ten. It is fitness for the use in front of you. The Government Data Quality Framework puts the question in a form that a delivery team can actually answer.

The quality dimensions you already know carry across, but their weighting changes. Completeness matters more, because a model cannot tell a missing value from a meaningful zero unless someone encoded the difference. Timeliness matters more, because an AI assistant is usually asked about the present. Consistency matters more, because two spellings of the same supplier become two suppliers in the eyes of the system. And a dimension that barely registers in reporting work, licensing, becomes decisive, because using material you are not permitted to use is not a quality defect that shows up in a dashboard.

There is a second reason the data side dominates, and it is structural rather than statistical. A model is retrained rarely and behaves the same way between retrainings. The data underneath it changes continuously, because source systems are upgraded, teams start populating a field they used to leave blank, a supplier changes how it codes a product, and a migration silently truncates a text column. An AI system is therefore a stable component sitting on a moving foundation, and almost all the observed variation in its behaviour over a year comes from the foundation. Monitoring the model without monitoring the inputs measures the part that is not moving.

Where the system is regulated, the obligations become explicit. ISO/IEC 5259 gives the international series for data quality in analytics and machine learning, and ISO/IEC 42001 gives the management system standard that puts an auditable process around AI. In the European Union, the AI Act sets data governance duties for high-risk systems specifically.

A vector embedding is a list of numbers that places a piece of text at a point in a space. The property that makes it useful is arranged during training: material about the same subject lands close together, and material about different subjects lands far apart. Nothing about the words is preserved directly. What is preserved is the neighbourhood.

The map above draws the idea at a scale you can see. Three tinted regions hold invoices, contracts and support tickets. A query is embedded by exactly the same process as the documents, so it becomes a point on the same plane, and the answer to the query is simply the set of points nearest to it. A search about a remittance concern lands inside the invoice region, and the dashed neighbourhood ring around it holds two chips, remittance and refund. Everything else falls outside the ring, and that includes the invoice chip itself as well as all of the contract and ticket material, which is the part worth pausing on: distance is measured from the query, not from the region the query happens to sit in, so belonging to a cluster is no guarantee of being returned by it. No keyword was matched either. The system found related wording by asking a distance question.

That is the whole trick, and it explains the behaviour analysts notice first: a search for staff turnover returns a document that says attrition, because attrition sits near turnover in the space even though the two strings share no characters. It also explains the failure that surprises people. A search for a part number or a surname can miss, because a rare code carries little of the contextual signal that positioned it, and the nearest neighbours of a code are often other codes rather than the document you wanted.

Three practical consequences follow, and each of them has bitten a delivery team. First, the diagram draws two axes and a real space has hundreds, so intuitions built from the picture about clusters being neatly separable do not transfer. Second, vectors produced by one model cannot be compared with vectors produced by another. They are positions in different spaces, and mixing them returns nonsense that looks like a working result. Third, changing the embedding model means re-embedding the entire corpus, which is a cost and an outage to plan for rather than a configuration change.

A vector database is simply a store that holds these positions and answers nearest-neighbour questions over them. What makes one choice better than another is a retrieval question rather than a representation question: how the index is built, whether its recall has been measured, and where the store sits relative to the access controls on the material it copied.

AI-ready data is a readiness test applied to a specific dataset for a specific intended use. It is not a category of data and not a file format. The phrase is often used loosely, so it is worth naming the document that gives it content in a UK public sector setting: the Government Digital Service and the Department for Science, Innovation and Technology published guidelines and best practices for making government datasets ready for AI on 19 January 2026.

That guidance carries a definition it takes from the Open Data Institute framework for AI-ready data, and the modal verb in it is doing real work. A dataset may be considered AI-ready when it addresses four components. It should be technically optimised, meaning structured and formatted for efficient use by machine learning systems and interoperability with AI tools. It should meet overall quality and adherence to standards, meaning accurate, complete, consistent and maintained through effective processes. It should be legally and regulatorily compliant, meaning aligned with applicable laws and regulations. And it should be responsibly managed, meaning managed ethically and securely throughout its lifecycle. Addressing the four is what makes readiness arguable for a stated use. It does not certify the dataset, and nothing in the guidance issues a badge.

The guidance then organises its own material around four foundations that overlap that list without matching it: technical optimisation; data and metadata quality; organisation and infrastructure context; and legal, security and ethical compliance. The third of those is the one summaries tend to lose, and it is the least technical. It is the organisational and infrastructure setting a dataset sits in, including the governance resources and working practices around it, which is to say whether anyone is funded and accountable to keep the dataset in the state the other components describe. A dataset can satisfy every technical test and still have nobody whose job it is to keep it that way next quarter.

The gate board above is not either of those lists. It is a reviewer’s working sequence: four questions that can be put to a candidate dataset in order, each with a visible answer. Are the records typed and readable. Does each field carry a recorded meaning. Does the licence allow this use. Can each record be traced back to a source. Structure is the practical face of technical optimisation and licensing is the practical face of legal compliance, but the other two gates are narrower than the components they serve. Documentation is one piece of evidence about quality, and provenance is one piece of evidence about responsible management. A dataset can clear all four gates and still fall short of the wider components, which is why the board is a triage instrument rather than a certification. The board draws both routes out of every gate, because the useful behaviour is not rejection but remediation: a dataset that fails on documentation gets its field definitions published and then re-enters the lane above the next gate rather than being waved through on the strength of having passed the first one.

Two of those gates hold reviews up, and neither is technical. Licensing questions surface material acquired years ago under terms nobody recorded, and the honest answer is sometimes that the source has to be dropped. Provenance questions surface datasets assembled by a person who has since left. The work of becoming AI-ready is therefore mostly documentation and permissions work, and that work lands on the data team rather than the modelling team.

At national scale, the same readiness problem is what the National Data Library programme is intended to address for UK public sector data, by arranging the sharing infrastructure that lets data be found and reused safely while keeping public confidence in how it is used.

A feature is a single input value a model reads: the number of orders a customer placed in the last ninety days, the average time between a meter reading and its submission, the count of failed logins in the past hour. Each one is the output of a calculation over raw data, and each calculation embeds decisions about windows, filters and how to treat missing values.

A feature store is the shared store for those values. It serves two audiences from one definition. An offline store holds the historical record used to assemble training data, so a training set can be built as of a past date. An online store holds a low-latency copy for live prediction, so a service can fetch the same feature in milliseconds while a request is waiting. Feast, one open source feature store among several, describes exactly this split, and describes the point-in-time correctness it generates as the way to avoid data leakage rather than debug error-prone joining logic by hand.

The failure it exists to prevent has a name. Training-serving skew is the gap that opens when the feature values a model learned from are not computed the same way as the values it is handed in production. It is common because the two calculations are usually written twice, once in a notebook against a warehouse table and once in application code against a live API, by different people months apart. The notebook counted orders over the last ninety days including today. The service counts over ninety days excluding today. Nothing errors. Every dashboard stays green. Accuracy simply degrades, because the model is now seeing inputs it never learned from.

The other property a feature store is bought for is time travel. Building an honest training set means asking what was known on the day each example occurred, not what is known now. If a customer later cancelled, their record today carries a cancellation the model would not have been able to see at prediction time, and training on it produces a system that looks excellent in evaluation and useless in production. Point-in-time joins are fiddly to write correctly and easy to get subtly wrong, which is a large part of why teams stop writing them by hand.

This is the same problem that a semantic layer solves for business metrics, arriving in machine learning clothing: one definition, computed once, reused everywhere. A team without a feature store is not necessarily wrong, but it does need another way to guarantee that the two computations agree, and a data contract on the feature definition is the usual substitute.

The Model Context Protocol is an open standard for connecting AI applications to outside systems: data sources such as files and databases, tools such as search or calculation services, and workflows. Its own documentation offers the analogy of a USB-C port for AI applications, and the analogy is precise about what is being standardised. A port settles the shape of the connection so that any conforming device can be plugged in. It says nothing about what the device is for or whether you should trust it.

Before a standard connector existed, every pairing of assistant and system needed bespoke integration work, so the number of integrations grew as the product of the two counts rather than their sum. That arithmetic is the economic argument for standardising the interface. The architectural argument matters more to a data professional. Once an assistant can reach a system through a standard connector, the thing that decides whether its answers are correct is no longer the integration. It is the access controls on the system behind the connector and the shared definitions that say what the returned fields mean.

The limits are where organisations get into trouble, and they are narrow enough to list. MCP standardises the interface. It does not confer permission, does not check whether the caller is entitled to the rows returned, does not reconcile two systems that define active customer differently, and does not create lineage for the answers it enables. Those remain governance responsibilities that sit with the systems being connected, and connecting an ungoverned system through a standard interface simply makes it easier to reach.