Quality engineering and observability
By the end of this module you will be able to:
- Turn a stage 1 quality dimension into an executable expectation with a threshold and an owner
- Distinguish testing, which asserts what you predicted, from observability, which detects what you did not
- Read and write the core of a data contract: schema, service levels, quality rules and owner
- Explain how a contract is enforced in continuous integration rather than in a policy document
The same silent change, found by a consumer or by a monitor
A silent change, where the meaning moves but the declared type does not, passes every assertion in both lanes, so the tested-only pipeline is repaired by a consumer after the harm lands and the observed one by the producer in the same run.
A test proves an assertion still holds; observability notices that the data stopped looking like itself. Run the same silent schema change down both lanes and the tested-only pipeline is repaired by a consumer after the harm lands, while the observed pipeline is repaired by the producer in the same run.
A data contract is one document holding five named parts
Schema, service levels, quality rules, ownership, and version with its deprecation window sit in one document because together they are the public interface consumers build against, which is why none of the five can move quietly.
A data contract is one document with five named parts: the schema, the service level objectives, the quality rules, the ownership, and the version with its deprecation window. Together they form the product's public interface, which is why none of them can change quietly.
The column that changed meaning without changing name
An operational team adds a new sales channel. To avoid a schema migration they reuse an existing status code, so a value that used to mean cancelled by the customer now also covers orders held for stock. Nothing about the column changes shape. It is still a short string, still never null, still drawn from a small set of allowed values.
Downstream, a churn model treats that status as a signal of customer intent, and a finance report excludes those orders from revenue. Both continue to run. The model quietly degrades over six weeks and the revenue figure drifts by an amount too small to notice in any single month. The first person to spot it is an analyst who cannot reconcile a total, four months after the change.
No assertion about the column could have caught this, because every assertion anyone had thought to write was still true. What would have caught it is a monitor on the distribution of values, which would have shown one code jumping from two per cent of rows to eleven overnight, and a contract that named a consumer to notify when the meaning of a field changes.
A field keeps its name, its type and its null rate, and every test still passes. What is left to catch the fact that it now means something different?
Stage 1 established what data quality means and how it is described. This module is about making those descriptions run. A quality dimension that lives in a policy document costs nothing and prevents nothing. The same dimension expressed as an assertion with a threshold, attached to a named dataset, owned by a named team and executed on every change is an engineering control. The distance between those two states is the entire subject here, and it is covered in four moves: expectations, observability, contracts and continuous integration.
A dimension names what good looks like. An expectation states the threshold, the column and the consequence of failing it.
27.1 From dimension to executable expectation
The Government Data Quality Framework sets out six core as defined by DAMA UK: completeness, uniqueness, consistency, timeliness, validity and accuracy. It also frames quality overall as fitness for purpose rather than perfection, which is the release valve that stops quality work becoming infinite. A dataset does not need to be flawless. It needs to be good enough for the decisions taken on it, and someone has to say what good enough means.
Turning a dimension into something a machine can check requires three additions. The first is a subject: which dataset and which column. The second is a threshold: not "complete" but "no more than 0.5 per cent of rows have a null postcode over a daily batch". The third is a consequence: does a failure block the release, quarantine the batch, or raise a warning that someone triages during working hours. A rule with no consequence is documentation.
An is the unit that carries all three. Great Expectations defines it as a verifiable assertion about data, comparable to an assertion in a unit test, and lets you state explicitly the condition the data should meet. Expectations are grouped into a suite, run against a defined batch of records, and return a pass or fail together with the offending values, which is what turns a complaint into a defect somebody can fix.
Worked example. Timeliness is the dimension. The dataset is the orders table. The expectation is that the maximum order timestamp is within ninety minutes of the run time. The threshold is ninety minutes at the ninety-fifth percentile over a rolling seven days. The owner is the orders platform team. The consequence is that a breach blocks publication of the downstream revenue mart and pages the on-call engineer. Six lines of specification, all of them testable, none of them a matter of opinion.
is the practice of running these assertions and failing the run when they are not met, in the same way unit tests are run against code. The assertions cover values, ranges, uniqueness, null rates and row counts rather than function return values. For analytics code written in , the tests live in the same repository as the models they guard, which means a change to a transformation and a change to the rule that protects it arrive in the same review.
“There are six core data quality dimensions, as defined by DAMA UK.”
The Government Data Quality Framework (GOV.UK) - Data quality dimensions
The framework gives UK public bodies a shared vocabulary, which matters because a quality conversation held without one collapses into taste. The dimensions are the naming layer only. They do not set thresholds, and the same framework treats quality as fitness for purpose, so the threshold has to come from the use, not from the standard.
Assertions catch the failures somebody imagined in advance. The incidents that damage trust are usually the ones nobody imagined.
27.2 Testing and observability are different jobs
Testing is deterministic and predefined. Someone decided that this column should never be null, wrote that down, and now every run checks it. The strength of testing is precision: a failure points at one rule, one column and one batch, and the fix is usually obvious. The limitation is equally clear. A test can only catch a failure that somebody predicted, and the space of ways data can go wrong is much larger than the space of ways anyone thought to write down.
works the other way round. Rather than asserting a condition, it watches the data continuously and looks for behaviour that departs from the recent past. Four signals carry most of the value: freshness, meaning when the table last received data; volume, meaning how many rows arrived compared with the usual pattern for that hour and weekday; schema, meaning whether columns appeared, disappeared or changed type; and distribution, meaning whether the shape of the values moved.
The opening story is the case observability exists for. Every assertion still passed, because the column kept its name, type and null rate. What moved was the distribution of one categorical value, from two per cent of rows to eleven, and no assertion existed that would have noticed. A distribution monitor flags that within a day without anyone having anticipated the change.
The failure modes are complementary, and so are the costs. Tests are cheap to run and expensive to think of. Monitors are cheap to think of and expensive to tune, because an anomaly detector with no seasonal awareness will fire every Monday morning and every bank holiday, and a team that has learned to ignore its alerts is worse off than a team with none. Observability is the layer above testing, not a replacement for it, and a platform that has bought a monitoring tool instead of writing assertions has bought the expensive half and skipped the cheap one.
One further boundary is worth naming. Neither testing nor observability can tell you whether a number is correct in the world. Both check that data behaves as data normally does. Accuracy against reality still requires reconciliation against an independent record, a physical count, or a second system, and no amount of pipeline instrumentation substitutes for that.
Common misconception
“If all our data tests pass, the data is good.”
Passing tests means no predicted failure occurred. It says nothing about failures nobody predicted, and nothing about whether the values match reality. A column can keep its name, type and null rate while its meaning changes, and every assertion will still pass. Testing is the floor of quality engineering. Continuous monitoring of freshness, volume, schema and distribution sits above it, and reconciliation against an independent source sits beside both.
Rules only bind when someone has agreed to them. A contract is where the promise, the threshold and the owner are written in one machine-readable place.
27.3 The anatomy of a data contract
A is a written, machine-readable agreement between the team producing a dataset and the teams using it. Because it is a file rather than a slide, it can be version controlled, reviewed in a pull request and checked automatically in the build. That single property is what moves a quality failure from the consumer's dashboard, where it is discovered late and cheaply blamed, to the producer's pull request, where it is cheap to fix.
The Open Data Contract Standard, maintained under the Linux Foundation, gives the shape. Its sections cover fundamentals, schema, references, data quality, support and communication channels, pricing, team, roles, service-level agreement, infrastructures and servers, and custom properties. Not every dataset needs every section. Four are effectively mandatory for anything another team depends on.
The first is : the fields, their types, which are required and what values are allowed. Without a declared schema every consumer rediscovers the shape by trial and error. The second is the quality rules: the expectations from section 27.1, expressed against named columns with thresholds. The third is the section, which in the standard covers properties such as availability, retention, frequency of update, latency, throughput, error rate, time to detect an issue, time to notify and time to repair. The fourth is ownership: a named team, a support route and a response expectation, because a contract with no counterparty is a wish.
The section teams most often omit is change policy, and it is the one that causes the incidents. A is any change that stops existing consumers working as they did: removing a column, renaming one, tightening a type, or changing what a field means. Adding an optional field is not one. Removing a field always is. Naming the change as breaking is what forces the conversation with consumers before the release rather than during the incident, and versioning the contract is what lets two consumers migrate at different speeds.
Contracts also need an escape valve. A rule that cannot be met is worse than no rule, because teams learn to route around the gate. The mature pattern is a documented waiver with an expiry date and an owner, recorded in the same repository, so that temporary tolerance of a known defect is visible rather than achieved by quietly deleting the check.
“This document describes the keys and values expected in a YAML data contract, per the Open Data Contract Standard.”
Open Data Contract Standard v3.1.0 (Bitol, Linux Foundation) - Executive summary
The word that carries the weight is YAML. A contract expressed as a structured file can be diffed, reviewed, versioned and executed by a pipeline, which is what separates it from the service-level appendix in a supplier agreement that nobody has opened since signature.
A contract enforced by goodwill is a document. A contract enforced by the build is a control.
27.4 How contracts run in continuous integration
applies the same mechanics that software teams use for code. When a pull request is opened against the analytics repository, an automated job builds the changed models and the ones downstream of them in a temporary schema unique to that request, runs the tests against the result, and posts the status back onto the pull request before anyone can merge. The dbt documentation describes exactly this shape, including building only the modified assets and their downstream dependencies rather than the whole warehouse.
Four checks belong in that job, and they run in order of cost. Contract linting comes first: is the contract file valid against the standard, does every column have a type, does every quality rule name a threshold. The Data Contract CLI covers this directly, with a lint command that validates one contract file and a changelog command that takes the published contract and the proposed one and reports what moved between them. Schema comparison comes second: does the proposed schema differ from the published contract, and if so is the difference additive or breaking. Rule execution comes third: do the expectations pass against the sample or the temporary build. Impact reporting comes last: which downstream models, dashboards and consumers touch the changed columns.
That last check depends on lineage. is an open standard for lineage metadata that pipelines and processing engines emit events to as they run, so the graph reflects what the jobs actually did rather than what a diagram said they would do. is the resolution that matters here: table-level lineage tells you that a report reads this table, while column-level lineage tells you whether the column you are about to change is one that report actually uses. That is the difference between blocking every change and blocking the ones that break something.
Two design decisions determine whether any of this is tolerated by the people subject to it. The first is where the gate sits. Blocking a merge is appropriate for a breaking schema change to a governed dataset and disproportionate for a warning-level null-rate drift, which belongs in a triage queue. The second is speed. A check that takes forty minutes will be bypassed within a quarter, so the pull request job must build only what changed and defer heavy validation to a scheduled run.
Finally, the enforcement point has to sit with the producer. Quality checks that only run in the consumer's pipeline detect the problem after it has been published, which is late and adversarial. Running the same contract at the point of production turns quality from an argument between teams into a property of the release.
A team states in its data quality policy that the customer table must be complete. An auditor asks how this is enforced. What is missing before the statement can be enforced at all?
A supplier changes the unit of a weight field from kilograms to pounds without renaming the column. The type is unchanged, no nulls appear, and every existing test passes. Which control is most likely to surface this quickly?
A producer wants to remove a column from a published dataset. The contract is in the repository and lineage is collected. What should the continuous integration job do?
Core distinctions
- The Government Data Quality Framework names six core dimensions defined by DAMA UK and treats quality as fitness for purpose, so the threshold has to come from the use rather than from the standard.
- A dimension becomes executable when it gains a subject, a threshold with a measurement window, an owner and a consequence. An expectation is the unit that carries all four.
- Testing asserts conditions somebody predicted and points precisely at what failed. Observability watches freshness, volume, schema and distribution for behaviour nobody predicted.
- Neither testing nor observability establishes that a value is correct in the world. Accuracy still needs reconciliation against an independent record.
- A data contract is machine-readable so it can be diffed, reviewed and executed. Its load-bearing sections are schema, quality rules, service levels, ownership and change policy.
- Removing or renaming a field is a breaking change and adding an optional one is not. Classifying the change is what forces the conversation before the release rather than during the incident.
- Contracts become controls when a pull request job builds only what changed, runs the rules, uses column-level lineage to find affected consumers, and gates proportionately at the producer rather than the consumer.
Standards and sources cited in this module
The Government Data Quality Framework (GOV.UK)
Data quality dimensions and fitness for purpose
UK public sector source for the six core dimensions defined by DAMA UK and for treating quality as fitness for purpose rather than perfection.
Open Data Contract Standard v3.1.0 (Bitol, Linux Foundation)
Executive summary and section structure
Defines the machine-readable contract shape used here: fundamentals, schema, data quality, service levels, team, roles and support.
Great Expectations Core documentation
Expectations, suites, validation definitions and checkpoints
Source for the definition of an expectation as a verifiable assertion about data and for how suites are validated against a batch.
dbt documentation, continuous integration jobs
CI jobs and slim builds
Describes building and testing only the modified assets and their downstream dependencies in a temporary schema, with status reported back to the pull request.
Specification, facets and column lineage
Open standard for lineage metadata emitted by running jobs, including the column lineage facet used for impact analysis.
Data Contract CLI documentation
Commands: lint and changelog
Source for the contract checks simulated here: lint validates a single contract file, and changelog compares a published contract with a proposed one.
Module 27 of 52 · Engineering and platforms