Orchestration and DataOps

Cron answers one question: what should start at this time. That is genuinely useful, and for a single independent job it is sufficient. Orchestration answers a different set of questions, and each of them is one that a schedule cannot express.

• Dependency, not timing. The orchestrator knows that aggregation runs after transformation because transformation produces what aggregation reads, rather than because one is set for four and the other for five. When the upstream step runs late, the downstream step waits instead of processing stale input.
• State that survives the run. The orchestrator records that this run, for this logical date, reached this step and failed there with this message. Cron records an exit code in a log file that nobody reads.
• Retry, backfill and catch-up. A failed step can be retried with a policy rather than by hand. A range of missed dates can be rerun as a first-class operation. A pipeline that was switched off for three days can be brought back up to date deliberately rather than accidentally.
• Observability of the run itself. How long each step took, how that compares with its own history, which step is on the critical path, and which downstream consumers are still waiting.

Underneath all of this sits one structure. A DAG is a set of tasks with arrows showing which must finish before which, and no path that loops back on itself. The acyclic property is what guarantees a run terminates and what lets the orchestrator work out a safe running order and what may run in parallel. It is also what makes a failure traceable: every task has a known set of predecessors, so the first failing step is the cause and everything after it is a symptom.

Two properties matter more than any orchestrator feature, because without them retries and backfills are dangerous rather than useful. A task should be idempotent, meaning that running it twice for the same logical date leaves the same result as running it once. And it should be partitioned by that logical date, so a rerun replaces one day rather than appending a second copy of it. A task that appends without a partition key will double the day it reruns, and the retry policy that was meant to protect the pipeline becomes the thing that corrupts it.

Apache Airflow 3.0 was released on 22 April 2025. The project's own release announcement puts its reach at over 30 million monthly downloads and 80,000 organisations, which is why a change to Airflow is worth reading as a statement about the category rather than as one vendor catching up.

1. DAG versioning. A run is now tied to the version of the code it started with, so a run completes on the definition it began with even if a new version is uploaded while it is in flight. Before this, editing a pipeline during a long run could leave a single execution half-built by two different definitions, which made a failure very hard to reason about after the fact.
2. Event-driven scheduling. Airflow can react to events happening outside Airflow, including data assets being created or updated by external systems. Work can therefore be triggered by data becoming ready rather than by the clock alone, which is what removes the reliance on a timing margin that the retailer above never knew it had.
3. Data assets. The unit the platform reasons about shifts from the task towards the dataset the task produces. A downstream pipeline declares the assets it consumes, and the orchestrator schedules it when those assets are refreshed. This is the same idea as a dependency graph, expressed in the vocabulary of the data rather than of the jobs.
4. A task execution interface. A client and server split that allows tasks to execute in other environments and other languages, so the orchestrator stops being the thing that has to host every dependency your transformations need.

The direction of travel across all four is the same. The orchestrator is becoming less of a timetable and more of a graph of data assets that knows what is fresh, what depends on it, and which version of the code produced it. If you are choosing an orchestrator today, that is the shape to compare against, whether the product is Airflow, Dagster, Prefect or a managed cloud service.

dbt is a transformation framework in which each model is a select statement in a file, and the tool takes care of materialising it as a table or a view in the warehouse. It matters less as a product than as a shape, because that shape is now what most warehouse transformation looks like whether or not the tool itself is used.

The idea that carries the weight is that models refer to each other by reference rather than by hard-coded table name. Because every dependency is declared in the code, the tool can derive the DAG from the project itself instead of asking an engineer to maintain a separate graph. That single decision produces four things that used to be separate projects.

• A build order for free. The graph is inferred, so the correct order and the safe parallelism come from the code rather than from a document that drifts.
• Data testing in the same repository. Assertions live beside the model they test. dbt ships four generic tests out of the box, checking that a column is unique, that it is not null, that its values come from an accepted list, and that a foreign key actually resolves to a row in the referenced model. Anything more specific is written as a select statement that returns the failing rows, so a test passes when it returns nothing.
• Documentation generated from the project. Descriptions written against models and columns are published as a browsable site, so the documentation is produced from the same commit as the code rather than maintained separately and left behind.
• Lineage as a by-product. Because dependencies are declared, the tool can answer what feeds this model and what breaks if I change it, which is exactly the impact analysis question that teams without lineage answer by asking around and hoping.

The wider point outlives any one tool. Transformations written as reviewed, tested, version-controlled code behave like software. Transformations written as ad hoc statements executed against production behave like an accident waiting for an audience. That discipline is what DataOps names.

CI/CD for data applies continuous integration and delivery to transformation code. When a pull request is opened, the changed models and the models downstream of them are built and tested in a temporary schema, and the result is reported back on the request before anyone merges. Delivery then promotes that same reviewed code onward through the environments and gates described below.

The environments come first, because everything else depends on them. At minimum you need three: a development environment where each engineer builds into their own schema and can break things freely, a staging environment that runs against production-shaped data on the production schedule, and production itself. The same code runs in all three; only the target schema and the credentials differ. If a change can only be tested by running it in production, you do not have a deployment process, you have a habit.

Building only what changed and what depends on it is the practical trick that makes this affordable. Rebuilding an entire warehouse on every pull request is too slow and too expensive to survive contact with a real team, so the pipeline uses the same dependency graph that produced the build order to work out the smallest set of models that could possibly be affected, and tests that.

Three gates then stand between a change and production.

1. The development gate. The schema difference is reviewed by a human who owns the model, and the unit-level assertions pass. This is where a breaking change is named as one: removing a column, renaming one, tightening a type, or changing what a field means. Adding an optional field is not breaking; removing a field always is.
2. The staging gate. The change runs against realistic volumes, and row-count and freshness assertions are checked. Logic that was correct on ten rows of sample data frequently is not correct on ten million rows with the awkward cases included.
3. The production gate. A breaking change needs a migration plan and a named owner signing it off, because at this point the cost of the change falls on consumers rather than on the team making it.

A data contract is what turns that third gate from a conversation into a check. Because the contract is a machine-readable file describing schema, quality expectations, service levels and ownership, the build can compare a proposed change against it and fail the pull request rather than waiting for a consumer to notice in production.

An orchestrator knows which tasks ran. A transformation tool knows which models depend on which. Neither on its own can tell you that the finance dashboard reads a column you are about to rename, because the chain crosses tool boundaries. Runtime lineage closes that gap by having each tool emit what it read and what it wrote as it runs, into a shared format.

Column-level lineage is the version worth insisting on. Table-level lineage tells you that a report reads this table, which in a wide table is nearly always true and therefore nearly useless. Column level tells you whether the specific field you are changing is one that the report actually consumes, which is the difference between a change request that takes an afternoon and one that stalls for a fortnight while somebody checks.

The modern data stack was assembled from specialists. One vendor moved data in, another transformed it, a third pushed results back out to operational systems, a fourth catalogued it. Between 2025 and 2026 those layers were pulled together. On 1 May 2025 Fivetran, an ingestion vendor, announced that it had signed an agreement to acquire the reverse-ETL company Census, subject to closing conditions. On 13 October 2025 Fivetran and dbt Labs announced an all-stock merger, which completed on 1 June 2026, placing ingestion, transformation and activation inside one company alongside a rewritten dbt execution engine named Fusion.

It is not obvious that this is bad. Fewer seams means fewer places where lineage is lost and fewer integrations to keep alive. But the buying position changes. When ingestion, transformation and the semantic layer are one product, the cost of leaving is no longer the cost of replacing one component; it is the cost of replacing the whole chain at once. The professional response is neither to refuse consolidated tools nor to hand over the whole estate, but to keep the exit cheap by making the parts that matter conform to open specifications.

Four hedges do most of that work, and all four are things you can require in a procurement.

• An open table format for storage. If the tables are Iceberg in object storage you own, the data survives a change of engine because the engine was never where the data lived.
• OpenLineage for lineage. Lineage emitted to a standard specification is portable. Lineage held in a vendor graph is not, and it is usually the first thing you lose in a migration.
• The Open Data Contract Standard for interfaces. Contracts expressed in a platform-agnostic file describe the promise between producer and consumer independently of whoever is currently executing it.
• An open metric specification for the semantic layer. The Open Semantic Interchange initiative, launched on 23 September 2025, exists so metric definitions can be written once in vendor-neutral form rather than re-encoded in each tool that consumes them.

None of these is free. Each adds a specification to learn and a conformance requirement to enforce. The question to ask of any one of them is what it would cost to move off this vendor in three years, and whether that number is one you would be willing to show the person who signs the contract.