Catalogues, lineage and active metadata

A business glossary is about language. It is the agreed list of terms an organisation uses, each with one written definition, so that active customer or working age means the same thing in a report, a contract and a model. Its unit is a term. Its owner is usually a business function rather than a platform team. Its success measure is that arguments about numbers stop being arguments about words.

A data catalogue is about assets. It is a searchable inventory recording what each dataset is, who owns it, where it sits and how far it can be trusted. Its unit is a dataset. Its success measure is that somebody can find and safely reuse a dataset without asking a colleague first.

Underneath both sits a third artefact that gets less attention. A metadata registry is a managed register in which the definitions of individual data elements are recorded, identified and administered, so that a field can be looked up rather than reinvented per project. ISO/IEC 11179 is the international series for metadata registries, and the framework part sets out how the rest of the series fits together. Throughout that series, metadata means descriptions of data rather than the data itself. Where a glossary settles a business term and a catalogue lists a dataset, a registry pins down the element: its name, its meaning, its permitted values and who is accountable for it.

The interoperability layer for catalogues is DCAT. It became a W3C Recommendation in its third version on 22 August 2024, and it exists so that catalogues published by different organisations can be read and aggregated without bespoke translation for each one.

The five-field catalogue entry figure at the head of the module sets a minimum shape. Five fields carry the load: title and summary saying what the dataset is and why it exists, columns and types describing its structure, a named accountable role rather than a team mailbox, a refresh schedule saying when the data updates, and a licence stating the reuse terms. Drop any one of those and the consumer has to ask somebody, which is exactly the outcome the catalogue was bought to avoid.

Active metadata is metadata that keeps itself current by reading signals from systems as they run, rather than waiting for someone to update a description by hand. The distinction is not about how modern the tool is. It is about where the fact comes from.

Declared metadata is asserted by a person: this table refreshes daily, this column holds a customer identifier, this dataset is owned by the pricing team. It is true on the day it is written. It becomes false silently, because a pipeline change does not send anyone to edit a catalogue entry. Declared metadata therefore has a half-life, and in most organisations that half-life is a matter of months.

Active metadata is emitted by the running system: the job that last wrote this table finished at 03:14, this table has been read by 6 queries this week and by 400 last quarter, this column was produced by that expression in that transform, this dataset has not been written for 11 days against a stated daily schedule. None of that requires a person to remember anything, and all of it changes the catalogue entry from a claim into an observation.

Three consequences follow. First, staleness becomes detectable rather than discovered: a freshness signal that contradicts a declared schedule is an alert, not an embarrassment six months later. Second, usage signals give you a ranking, so a search for customer can put the table 400 people query above the thirty-nine nobody touches. Third, active metadata is what makes data observability and the catalogue the same system rather than two, because the freshness, volume and schema signals an observability tool watches are the same signals the catalogue needs to stay honest.

Data lineage is the record of where a piece of data came from and what happened to it on the way, from source system through each transformation to the report someone is looking at. Lineage drawn by hand on a whiteboard is documentation and rots like documentation. Lineage captured at run time is evidence, because it is produced by the same execution that moved the data.

OpenLineage is the open standard for that capture. Rather than each tool inventing its own lineage format, integrations in orchestrators, engines and warehouses emit events in a common shape as jobs start and finish, and any compatible backend can consume them.

Table-level lineage tells you that a report reads a table. Column-level lineage tells you whether the column you are about to change is one that report actually uses. The OpenLineage column lineage dataset facet records, for each output column, which input columns produced it and in what way, and it distinguishes a direct dependency, where the output value is derived from the input, from an indirect one, where the input shaped the result through a join, a filter or a sort without being derived into it.

That distinction is the difference between a useful answer and a useless one. A filter predicate on a column means changing that column can change which rows appear, even though the column never shows up in the output. A team that only tracks direct derivation will miss that case and ship a change that silently alters row counts downstream.

Impact analysis means working out what depends on something before you change it, by following lineage downstream from the column, table or job in question. Read the other way, upstream from a report that has gone wrong, the same graph is how you find the cause.

Work through the lineage impact graph at the head of the module. The proposed change edits order_total in orders_raw. Nothing changes in customers_raw. Both feed the build_sales transform, which reads order_total, so the transform is in scope. From there the graph separates. The sales_daily mart sums order_total and is affected. The revenue_by_sku mart uses order_total and is affected. The customer_counts mart does not reference order_total at all and is not. Following the dashboards, the trading review reads the affected marts and needs its totals rechecked; customer growth reads counts only and needs nothing.

The result is a short list: one transform, two marts, one dashboard, with a named owner for each and a specific question to answer. The alternative, in an organisation without column-level lineage, is a change freeze while everyone checks everything, or a change shipped in hope. Teams without lineage do not usually make reckless changes. They quietly stop making changes at all, and the debt accumulates where nobody can see it.

A breaking change is one that stops existing consumers working as they did: removing a column, renaming one, tightening a type or changing the meaning of a value. Impact analysis turns that from a category into a list of names. Where a data contract exists between producer and consumers, the lineage graph tells you which contracts the change touches, and the contract tells you what notice the consumers were promised.

The catalogue trust loop figure at the head of the module has five steps and one fail route. An owner publishes an entry recording meaning, ownership and freshness. A reader searching before building discovers it. The entry answers the question actually being asked, so it gets used. The reader feeds back what was wrong or missing. The owner corrects it and dates the change, and the refreshed entry is published again. The loop holds only while all five steps keep circulating.

The fail route is what happens when a reader searches and the entry does not answer the question: the catalogue is bypassed and the reader asks a colleague instead. That is the honest statement of the adoption battle. The catalogue is not competing with another catalogue. It is competing with a knowledgeable colleague who responds quickly and is usually right. Beating that requires the catalogue to be faster than a message and at least as trustworthy, and a catalogue full of stale or unowned entries is worse than no catalogue, because it teaches people to distrust the one place that was meant to settle questions.

Four design choices follow from taking that competitor seriously.

1. Curate a subset rather than harvest everything. A hundred entries that are owned, described and current beat forty thousand that are not. Harvest for coverage if you like, but rank and surface only what somebody has signed for.
2. Name a person, not a team mailbox. A data steward is the named person who looks after a defined set of data day to day, agreeing what fields mean, watching quality and chasing issues to closure. An entry without one is an entry nobody will correct.
3. Make feedback cost one click. The loop only closes if flagging a wrong entry is easier than working around it. Feedback that requires a ticket will not happen.
4. Let runtime signals do the maintaining. Freshness, usage and column lineage should arrive from the platform. Reserve human effort for the parts a machine cannot supply: what the dataset is for, what it must not be used for, and who decides.

One further honesty is worth adding. Catalogue adoption is measured badly almost everywhere. Entry counts and coverage percentages measure the harvest, not the outcome. The measures that mean something are how often search leads to a dataset being used, how many entries have been corrected in the last quarter, and whether the median age of an entry is falling.