Metadata and context

"Data about data" is the definition everybody repeats, and it explains nothing. It tells you that metadata is a kind of data, which you already knew, and leaves you no test for deciding whether a particular field belongs in the record. It is also circular in practice: one team's metadata is another team's data. The refresh timestamp on a warehouse table is metadata to the analyst reading the table and operational data to the platform team measuring pipeline reliability.

A definition with teeth works from the reader rather than from the object. Metadata is the recorded answer to the questions a competent stranger must resolve before they can use a dataset safely: what is this, how is it arranged, where did it come from, how current is it, and what am I allowed to do with it. If a field answers one of those questions, it belongs in the record. If it does not, it is either payload or decoration.

The word "stranger" is doing real work in that sentence. Metadata is not written for the person who built the dataset, because that person already holds the answers in their head. It is written for the analyst three teams away, for the auditor two years later, and increasingly for a machine that has to decide whether a dataset is fit for a purpose without a human in the loop. The test of a metadata record is not whether it looks complete. It is whether someone with no access to the original team can act on the data and be right.

This is why metadata is described as an asset rather than as paperwork. A dataset whose meaning cannot be reconstructed is not an asset at all: it cannot be published, shared, joined, audited or safely deleted, because none of those actions can be justified without knowing what the data is. Naming something a data asset is a claim that someone owns it and manages it, and that claim is only credible if the surrounding record exists.

The figure above stacks the three layers over a bare table of values, and the arrangement is the argument: the values at the bottom answer nothing on their own, and each layer above adds a class of answer the values cannot supply.

1. Structural metadata says how the data is arranged. Field names, data types, units, precision, allowed values, primary keys, the relationship between one table and another, the order of columns in a file, the character encoding. Structural metadata is what a schema makes formal and machine-checkable, and it is the layer most often already present in some form, because a database or a file format usually insists on it.
2. Descriptive metadata says what the data is. Title, plain-language description, subject or theme, keywords, the period the data covers, the geographic area it covers, the population it was drawn from, known limitations. Descriptive metadata is what makes a dataset findable by someone who does not already know it exists, and it is the layer most often missing, because it cannot be inferred from the file.
3. Administrative metadata says who is answerable for the data and on what terms it may be used. Publisher, owner, steward, licence, access conditions, lawful basis for personal data, retention period, update frequency, version, date created, date last modified. Administrative metadata is what turns a file into something an organisation can be accountable for.

The three-way split matters because the three families have different owners, different failure modes and different half-lives. Structural metadata is usually produced by engineers and goes stale the moment a schema changes without anyone noticing. Descriptive metadata is usually produced by the subject expert and is stable for years. Administrative metadata belongs to whoever is accountable for the data, changes when ownership or licensing changes, and is the layer that regulators ask about first. A record that is strong in one family and empty in another fails in a predictable way: strong structure with no description gives you a dataset nobody can find, and strong description with no administration gives you a dataset nobody is allowed to use.

One useful habit when reviewing a record is to tag each field with the family it belongs to and then count. A catalogue entry that is nine parts descriptive and administrative with one or two structural fields is normal and healthy, because the deep structural detail lives in the schema rather than in the catalogue. A catalogue entry that is entirely structural is a table listing pretending to be a catalogue.

Take the substation file from the opening. A minimally useful record is not a hundred fields. It is the smallest set that lets a stranger use the file correctly, and for a tabular file that set is short enough to write in twenty minutes.

• Title and description. Not the filename. A sentence saying what the rows are: one row per substation per half-hour settlement period, half-hourly demand readings for primary substations in the North West licence area.
• Coverage.The period the data covers and the population or area it covers, including the gaps. "January 2024 to December 2025, excluding March 2024 where the telemetry feed was down" is worth more than a clean range that quietly hides a hole.
• A field dictionary. One line per column giving the name, the type, the unit, and the meaning. This is where column G stops being a mystery. For any coded column, either the code list or a pointer to the reference data that defines it.
• Provenance. The source system, how the extract was produced, when it was taken, and any transformation applied on the way, including filtering and rounding. An extract that silently dropped rows is a different dataset from the one its title claims.
• Accountability and terms. Who owns it, who to ask, the licence or internal access conditions, whether it contains personal data, and the retention period. If it holds personal data and no lawful basis is recorded, that is a finding rather than an omission.
• Currency. Version, date produced, and whether it will be updated again. A file that will never be refreshed should say so, because a stale dataset described as live is more dangerous than one honestly labelled as a snapshot.

Notice how much of that list is administrative and descriptive, and how little is structural beyond the field dictionary. Notice also that most of it is known on day one and none of it is derivable from the file itself. That asymmetry is the whole argument for capturing metadata at creation.

A second habit worth forming is to write the definitions somewhere the organisation can reuse them. If demand is defined in the field dictionary of one spreadsheet, the next spreadsheet will define it again and slightly differently. A shared business glossary holds one written definition per term so that the same word means the same thing in a report, a contract and a model, and the field dictionary then points at it rather than restating it.

The moment two teams describe datasets independently, they invent two field names for the same idea. One writes owner, the other writes publisher, a third writes data_custodian, and a search across all three finds nothing. Shared vocabularies exist to stop that.

DCAT, the W3C Data Catalog Vocabulary, is the widely used vocabulary for describing datasets, the services that serve them, and the catalogues that list them. Taken field by field, a single catalogue entry maps almost entirely onto descriptive and administrative properties, with only a couple saying how the bytes are arranged. Several of the property names come from Dublin Core terms, which DCAT reuses rather than duplicating, so a title is dcterms:title and a licence is dcterms:license. The specification binds the prefix dcterms: to the Dublin Core terms namespace in its own namespace table; the shorter dct: turns up widely in examples and tooling, but it is not the prefix the specification itself uses.

Where a vocabulary standardises the names of fields, a metadata registry standardises the definitions of the data elements themselves. ISO/IEC 11179 provides the framework for this: a managed register in which each data element is recorded, identified and administered, so that anyone using a field can look up what it means and how it is represented rather than guessing from its name. The distinction is worth holding onto. DCAT helps you find the dataset. A registry helps two datasets agree on what customer status actually means before anyone joins them.

The same discipline scales down. Inside a single organisation, the catalogue entry, the glossary term and the registry definition are three views of one commitment: that meaning is written down once, in a place with an owner, rather than rebuilt from memory each time somebody needs it.

Four capabilities that organisations buy tooling for are made of metadata and nothing else. If the metadata is thin, they are thin, and no amount of tooling repairs that.

1. Catalogues. A data catalogue is a searchable inventory of what an organisation holds. Its search index is descriptive metadata, its access decisions are administrative metadata, and its trust signals are quality metadata. A catalogue populated by crawling schemas alone will list every table and explain none of them.
2. Lineage. Data lineage records where a value came from and what happened to it on the way. It is metadata emitted by pipelines at run time, and it is what makes impact analysis, debugging and audit evidence possible. A pipeline that emits no run metadata leaves no lineage to query.
3. FAIR. The FAIR principles ask that data be findable, accessible, interoperable and reusable, for machines as well as people. Most of the work falls on metadata: persistent identifiers, rich description, shared vocabularies and clear licensing. FAIR is a set of principles about machine-actionable description, and it is not the same thing as open data, which is a statement about permission to reuse.
4. Automation. Active metadata keeps itself current by reading signals from running systems rather than waiting for a person to update a description. Freshness checks, usage-based trust scores and automatic deprecation all depend on the platform emitting metadata continuously.

Each of those has the same dependency shape. The capability is built at a later stage, but the raw material is captured at the earliest one, by whoever creates the data. That is the practical reason metadata belongs in the foundations of a data course rather than in an appendix: the decisions that make catalogues, lineage and FAIR achievable are made long before anyone is thinking about them.