Training data, synthetic data and provenance

Training datais the collection of examples a model learns from. Its coverage, its errors and its biases become the model's behaviour, which is why the size of a corpus is a poor proxy for its value. Data curation is the active work of keeping that collection usable: describing it, checking it, deciding what to keep and preserving it against format and system change. For a training corpus, three specific operations carry most of the weight.

Deduplication removes exact and near-duplicate documents. Web crawls are heavily repetitive: the same article is syndicated across dozens of sites, boilerplate navigation text repeats on every page of a domain, and popular passages are quoted endlessly. Duplicates do three kinds of damage. They waste training compute on content the model has already seen. They increase the chance that the model memorises and reproduces a specific passage verbatim, which is both a copyright exposure and a privacy exposure. And they contaminate evaluation, because a benchmark question that also appears in the corpus turns a test of generalisation into a test of recall. Near-duplicate detection is usually done with hashing over shingles of text rather than exact string comparison, because the interesting duplicates differ by a header, a date or a paragraph.

Filtering decides what is admitted. Language identification keeps the corpus to the languages the model is meant to serve. Quality heuristics drop pages that are mostly link farms, machine-translated filler or repeated character runs. Safety filters remove categories the organisation has decided not to train on. Personal data filters attempt to strip identifiers that have no business being in a corpus at all. Every filter is a judgement with a cost on both sides: filter too little and the model learns the sludge, filter too hard and you remove the minority dialects, technical registers and rare topics that made the corpus worth collecting.

Licensing establishes whether you are entitled to use what you have collected. This is where many corpora fail an audit, because the crawl came first and the rights question came second. A defensible corpus records, per source, the licence or legal basis relied on, the date the material was collected, whether a rights reservation was expressed by the publisher, and what the organisation did about it. Under EU law that includes reservations of the text and data mining exception expressed under Article 4(3) of Directive (EU) 2019/790, which providers of general-purpose models are expected to identify and respect.

The output of curation is not only a filtered corpus. It is a curation record: for each source, what was taken, on what basis, when, what was removed and why. That record is the difference between a team that can answer a regulator in an afternoon and a team that cannot answer at all.

Model collapse is the degradation that happens when models are trained repeatedly on content that earlier models generated. The model collapse loop at the head of the module has four turns. A model generates fluent text that is mostly drawn from the dense middle of what it learned. The web absorbs that text as pages, answers and summaries. A crawler collects the next corpus, which now contains that output. The next model trains on a copy of a copy.

The important detail is the order in which things are lost. Rare cases go first. The common middle of the distribution survives each pass reasonably well, because it is what the previous model was most likely to produce; the tails thin out, because unusual constructions, minority dialects, uncommon named entities and low-frequency facts were rarely sampled in the first place. Several generations in, the model still sounds fluent. It has simply stopped knowing about the unusual, and fluency is exactly the signal least likely to reveal that.

There is no architectural fix for this. A larger model trained on the same recursive corpus inherits the same thinning. The exit is sourcing you can name, licence and date: curated corpora whose origin is recorded, licensed material obtained under agreement, archives with a known collection date that predates or is isolated from generated content, and freshly captured data whose capture is attested. This is why content provenance stopped being a media-industry concern and became a data engineering requirement.

Synthetic data is records generated to resemble real data rather than collected from real people or events. Three uses stand up to scrutiny.

1. Test and development environments. Engineers need realistic data shapes to build and test against. Copying production records containing personal data into a development environment is one of the most common and least defensible privacy failures in a data platform. Generated records with the same schema, cardinality and edge cases remove the need.
2. Balancing rare classes. Where a genuine event is rare, such as a specific fraud pattern or an uncommon fault signature, generating additional examples can stop a model treating the rare class as noise. This is augmentation, and it works best when the generator encodes real domain structure rather than resampling the few examples available.
3. A lower-risk substitute for sharing. Where a dataset cannot be released because of sensitivity, a generated stand-in can let a partner develop and test an approach before applying it to the real data under controlled access.

The two failure modes are equally well established.

1. It cannot add information that was not there. A generator learns the distribution of its source. If the source under-represents a group, the generated data will under-represent it too, usually more sharply, because the generator is drawn towards the dense regions of what it saw. Generating a million synthetic records from a biased sample produces a million biased records and a false impression of statistical strength.
2. It is not automatically anonymous. A generator fitted closely to its source can reproduce real records or leak facts about them. A membership inference attack tries to determine whether a particular individual was in the source set using only the outputs, and a well fitted generator can make that determination easy. Synthetic data is a privacy-enhancing technique with an evidence bar, not a privacy guarantee, and the ICO treats it that way.

The practical rule is that synthetic data must be labelled as synthetic wherever it travels, and must never be silently merged into a corpus that is also used to measure real-world performance. The moment the label is lost, you have manufactured the first turn of the collapse loop inside your own organisation.

Content Credentials are the user-facing name for provenance data carried under the specification published by the Coalition for Content Provenance and Authenticity, known as C2PA. The C2PA describes its work as an open technical standard for publishers, creators and consumers to establish the origin and edits of digital content, and describes Content Credentials as working like a nutrition label for digital content.

The mechanism is a chain of signed claims rather than a single badge, which is the part most often misunderstood. At capture, a device can sign an assertion about what it recorded. At edit, a tool can add a signed assertion about what it changed. At publish, a platform can retain the accumulated record. Verification then walks the whole chain and checks each claim back to the signer that made it, not merely the last one. That is what makes the record tamper-evident: an alteration after a claim was signed breaks the signature over that claim.

Two conclusions follow, and only two. If every claim resolves to a signer, you have history you can show, which is a much stronger position than judging an image by eye. If no credential is present, nothing is established. Absence is not evidence that the asset was faked, because most files in circulation carry no credential at all and many pipelines strip metadata as a matter of routine. Treat an uncredentialed asset as unverified, not as proven false. A team that treats missing credentials as a positive finding of fabrication will make confident and wrong decisions.

Provenance does for images, audio and video what data lineage does for tables, and the two share a conceptual model. The W3C PROV data model gives the vocabulary that dataset-level provenance systems reuse.

Regulation (EU) 2024/1689 places data obligations in two distinct places, aimed at two distinct actors. Confusing them is a common error in compliance documents.

Article 10 applies to high-risk AI systems. It is about the quality and governance of the data sets used to build a specific system for a specific purpose. It requires data governance and management practices appropriate for the intended purpose, and it sets a standard for the data sets themselves.

Article 10 also contains a provision that is frequently misread as a prohibition. To the extent strictly necessary for bias detection and correction, providers of high-risk systems may exceptionally process special categories of personal data, subject to appropriate safeguards. That is a narrow permission with conditions attached, not a general licence, and not a requirement to collect such data.

Article 53 applies to providers of general-purpose AI models. The obligations there are about the model rather than a downstream system, and two points bear on data directly. Point (c) requires a provider to put in place a policy to comply with Union law on copyright and related rights, including identifying and complying with a reservation of rights expressed under Article 4(3) of Directive (EU) 2019/790. Point (d) requires the provider to draw up and make publicly available a sufficiently detailed summary about the content used for training of the model, according to a template provided by the AI Office.

The practical consequence of point (d) is that the curation record described earlier becomes an externally facing document. A team that only kept the filtered corpus, and not the record of how it was assembled, cannot produce that summary retrospectively at any reasonable cost. Alongside the regulation, ISO/IEC 42001 gives the management system shape for the surrounding controls, ISO/IEC 8183:2023 sets out a data life cycle framework for artificial intelligence, and the ISO/IEC 5259 series addresses data quality for analytics and machine learning.