Governance and stewardship
By the end of this module you will be able to:
- Compare DAMA and IBM governance maturity frameworks
- Describe the purpose and components of a data catalogue
- Explain the data contracts pattern and the problem it solves
- Design a basic lineage tracking approach for a described pipeline
Governance and stewardship as a four-step feedback loop
Governance closes the loop from policy to standards to stewardship to evidence and back to policy.
Governance is the loop: Policy sets intent, Standards translate it into measurable rules, Stewardship operates the rules, Evidence audits and feeds policy revision. Skipping evidence turns governance into a one-time exercise.
Five fields every catalogue entry must carry
A useful catalogue entry carries five fields: title, schema, owner, refresh schedule, licence.
A useful catalogue entry carries five fields: title + summary, schema reference, owner, refresh schedule, licence. Drop any one and consumers cannot use the dataset without asking the producer. W3C DCAT-3 describes catalogue metadata; schema and ownership conventions still need local governance.
Real-world pattern · ongoing
A renamed column can break a board report before anyone notices.
In many data platforms, a source team can rename a column, change a type, or change the meaning of a field without realising which dashboards, models, and regulatory extracts depend on it. The first visible symptom may be a failed board pack or a model trained on malformed data.
Data contracts address that failure mode by turning expectations into machine-checkable agreements between producers and consumers. The previous module covered how data moves through pipelines. This module covers how governance ensures those pipelines remain trustworthy.
A source team renamed a column. Downstream reports broke. Nobody noticed for three days. Could a formal agreement between teams have prevented this?
Governance without tooling is just policy. Governance with tooling becomes operational. Data catalogues make data discoverable. Data contracts make interfaces reliable. Lineage makes failures traceable. Together they form the infrastructure of trust.
Governance starts with knowing which data assets exist, what they mean, and who can make decisions about them.
13.1 Data catalogues
A data catalogue is an organised inventory of an organisation's data assets. It stores technical metadata (schema, column types, row counts), business metadata (descriptions, owners, stewards, quality scores), and operational metadata (freshness, lineage, access patterns). Modern catalogues such as DataHub, OpenMetadata, Collibra, and Alation can crawl sources automatically, but the business meaning still has to be governed.
Without a catalogue, analysts lose time searching for data, verifying its meaning, and confirming whether they are allowed to use it. A well-maintained catalogue reduces ambiguity by answering four questions quickly: what is this asset, who owns it, can I use it, and is it fit for this purpose?
Worked example: a customer table with 64 columns is not governed because it appears in a catalogue. It is governed when the catalogue identifies the accountable owner, the steward, the lawful-use constraint, the retention class, the quality rules, and the downstream assets that depend on it.
“Effective, efficient, and acceptable use of data”
ISO/IEC 38505-1:2017 - Abstract
ISO frames data governance as board-level direction over the use and protection of data. A catalogue is one operational control that supports that duty, but it is not governance by itself.
A catalogue says what the asset is. A contract says what consumers can rely on.
13.2 Data contracts
A data contract is a formal agreement between a data producer and one or more data consumers. It specifies the schema, semantic definitions, quality expectations, freshness commitment, owner, support route, and change policy for a data asset.
Contracts should be treated like interfaces, not policy PDFs. Automated checks run when data is published or changed. If a producer removes a required field, changes a type, or misses a freshness service level, the pipeline blocks, alerts, or routes the change for approval before the break reaches consumers.
The leadership implication is accountability. A contract makes it clear which team owns the commitment, which consumers depend on it, and what happens when the commitment is missed.
“Keys and values expected in a YAML data contract”
Open Data Contract Standard v3.1.0 - Executive Summary
ODCS makes the contract concrete: fundamentals, schema, quality, roles, service levels, infrastructure, and support can be expressed as structured YAML and checked by tooling.
Common misconception
“Data contracts add bureaucracy that slows teams down.”
Without contracts, schema changes silently break downstream pipelines. The cost of debugging silent failures can exceed the cost of maintaining a contract. Contracts shift the cost of change from consumers, who suffer breakage, to producers, who must communicate and test changes. This is not bureaucracy; it is engineering discipline.
When a control fails, lineage shows where the data came from and who is affected.
13.3 Lineage in practice
Module 9 introduced data lineage conceptually. In the Applied stage, lineage becomes operational. Tools and standards such as OpenLineage capture metadata about datasets, jobs, and runs by instrumenting pipeline tools such as Airflow, Spark, and dbt. Every time a transformation runs, the system records the inputs, process, and outputs.
Column-level lineage traces individual fields through joins, filters, casts, and aggregations. If a revenue figure in a board report is wrong, table-level lineage tells you which tables were involved. Column-level lineage shows which source field and transformation created the figure.
The practical standard is impact analysis. Before a producer changes a field, lineage should show the affected dashboards, extracts, models, and owners. After an incident, lineage should shorten diagnosis from days to minutes.
Common misconception
“We track lineage at the table level. That is sufficient.”
Table-level lineage shows that table A feeds table B. But when a specific column value is wrong, you need column-level lineage to identify which source field and which transformation produced it. Table-level lineage narrows the search; column-level lineage finds the answer.
An upstream team renames customer_id to customer_identifier without warning. Downstream dashboards fail the next morning. Which governance mechanism would most directly prevent this?
An analyst spends two hours searching for a customer dataset, verifying its meaning, and confirming access permissions. Which tool addresses this problem?
A data quality issue is reported in a monthly revenue figure. The data warehouse team uses table-level lineage to identify that the revenue table was populated from three source tables. They still cannot find which specific column introduced the error. What is missing?
Governance checks to carry forward
- A catalogue is governed metadata, not a search box. It must carry owner, steward, meaning, access route, quality signal, and usage constraints.
- A data contract is an enforceable interface between producer and consumer. It should cover schema, meaning, quality, freshness, ownership, support, and change policy.
- Lineage is useful when it supports impact analysis and incident diagnosis. Table-level lineage narrows the search; column-level lineage finds the field-level cause.
- ISO/IEC 38505-1:2017 places data governance at governing-body level, focused on effective, efficient, and acceptable use of data.
- Governance should be implemented in sequence: inventory, ownership, quality rules, contracts, lineage, then measured expansion.
Standards and sources cited in this module
ISO/IEC 38505-1:2017, Governance of data
Abstract
Primary standard for governing-body responsibilities over effective, efficient, and acceptable use of data.
Data Governance knowledge area
Industry guidance on governance structures, stewardship, and maturity models.
UK Data Standards Authority guidance (2023)
Full guidance
UK government approach to data standards and interoperability governance.
Open Data Contract Standard v3.1.0
Definition and schema sections
Open YAML standard for data contracts, including schema, quality, roles, service levels, and support metadata.
Project overview
Open standard and tool ecosystem for collecting lineage metadata about datasets, jobs, and runs.
Module 13 of 26 · Applied Data