Stage 3 summary. Engineering and platforms

Data engineering has a five-step shape: generation in the source systems, ingestion into the platform, storage, transformation, and serving to whoever or whatever consumes it. Generation almost always sits in systems the data team does not control, so the platform inherits their timing, outages and schema changes, while serving is where the value is finally claimed and every earlier step is judged by whether it made the last one trustworthy.

ETL transforms before the data lands in the destination; ELT loads it first and transforms inside the destination on that engine's compute. The decision is about cost and control, not fashion. ELT became common because elastic warehouse compute is often cheaper than a separate processing tier, and because a raw landing lets a team re-derive everything when a rule turns out to be wrong. ETL still wins where the destination is expensive to compute in, or where material must not land raw for legal reasons.

A step is idempotent when running it twice leaves the same result as running it once. Reruns are the normal case rather than an accident: a file arrives late, a node dies mid-write, a source issues a correction. So the design question is how to make a rerun uneventful, usually by replacing a whole partition deterministically rather than appending. Once that holds, a backfill is ordinary work: bound the range, push it through the daily job's own code path, and reconcile the periods that should not have moved.

Cron answers one question: what should start at this time. It cannot express that this job depends on that one having finished, and the failure that follows is quiet rather than loud. A retailer's nine nightly jobs each started on time, found the previous day's file because the extract ran late, and reported success for eleven days. Every job was green and every number was stale, because a timetable had been asked to carry a dependency.

Apache Airflow 3.0, released on 22 April 2025, reads well as a statement about where the category went. DAG versioning ties a run to the code it started with, so a definition edited mid-run no longer produces a half-built execution. Event-driven scheduling lets work be triggered by data becoming ready. Data assets shift the unit of reasoning from the task to what the task produces. A task execution interface splits execution from the scheduler, so the orchestrator no longer hosts every dependency. Less timetable, more graph of assets that knows what is fresh.

The transformations themselves became code. Where models are select statements referring to each other rather than to hard-coded table names, the tool derives the build order, runs assertions beside the models they guard, publishes documentation from the same commit and produces lineage as a by-product. DataOps is what happens between that code and production: three environments running the same code against different targets, changed models and their descendants tested in a temporary schema on every pull request, and a promotion gate somebody believes.

Batch against streaming is not a question about modernity. It asks how old the data may be when somebody or something acts on it, and what the wrong outcome costs. In a retailer most reporting is unharmed by seven-hour-old numbers, and the damage sits in a narrow band of decisions such as whether to show an item as in stock. Price those decisions and buy freshness only for them.

Underneath streaming sits the log: an append-only, ordered record that consumers read at their own position. A topic names a stream of those records, and a consumer group lets several processes share one topic's partitions without reading the same record twice. Apache Kafka 4.0, released on 18 March 2025, was the first major release to run entirely without ZooKeeper, making KRaft mode the default, so a cluster is one distributed system to operate rather than two. Share groups landed in the same release as an early access feature: a signal about direction rather than a foundation to settle payments on this year.

Most organisations do not emit events, they write database rows, and change data capture is how those rows become a stream. Polling a table for a modified timestamp loses deletes, misses intermediate states, depends on every writer maintaining the timestamp and applies analytical load to a transactional system; reading the database's own commit log avoids all four. The remaining question is what happens on a restart. At-most-once drops records, at-least-once duplicates them so consumers must be idempotent, and exactly-once holds only where the sink takes part in the guarantee.

ACID names four guarantees a transaction can make: it happens completely or not at all, it leaves the data valid against its rules, it is not corrupted by transactions running alongside it, and once committed it survives a crash. Those guarantees are paid for with coordination, which costs latency and constrains how far a system can spread, so buying them everywhere by reflex is as much an error as refusing them.

The CAP intuition is routinely overstated. It says only that when a network partition happens, a distributed system must choose between refusing to answer and answering with data that may be stale. It says nothing about the far more common healthy case, where the real trade is between latency and how strictly reads reflect the most recent write.

Choose the data model from the workload rather than the label. Relational fits data of stable shape queried by joining it and written transactionally, document fits records of varying shape usually read whole, key-value fits lookup by a known key at high rates, and graph fits work where the relationships are the query. An in-process engine such as DuckDB has then moved the boundary underneath all four: much of what was called big data was a cluster answer to a laptop-sized question.

The three platform shapes are best told apart by who operates what. A warehouse manages storage and engine together, giving strong guarantees in exchange for accepting its format and its bill. A lake is files in object storage that you own outright and that guarantee nothing on their own. A lakehouse puts an open table format over those files so they behave like warehouse tables while the storage stays yours.

The table format is the whole trick. Apache Iceberg and comparable formats add a metadata layer above columnar files, supplying atomic commits, schema evolution without rewriting history, snapshot reads and safe concurrent readers and writers. Once tables are open, the engine stops being the thing that holds the data, which is why the catalogue layer became the point of leverage between 2024 and 2026: it is where the decision about which engine may read which table, under whose permissions, now lives.

The major platforms are best compared as operating models rather than feature lists: how compute is billed and isolated, how governance is expressed, and what leaving would cost. DuckLake, zero-ETL services and reverse ETL each remove a piece of pipeline without removing the modelling, quality or contract work that made it meaningful. Medallion, with its bronze, silver and gold layers, is a naming convention for degrees of refinement, and calling a layer gold does not make anything in it true.

Stage 1 established what quality dimensions mean. Making one executable takes three additions: a subject, meaning this dataset and this column; a threshold, so not complete but no more than half a per cent of rows missing a postcode over a daily batch; and a consequence, deciding whether a breach blocks the release, quarantines the batch or raises a warning. A rule with no consequence is documentation with a schedule attached.

Testing and observability are different jobs and neither substitutes for the other. A test asserts something you predicted; observability watches the shape of the data for what nobody predicted. The clearest case is an operational team reusing a status code so one value quietly acquires a second meaning: every assertion anyone had written stayed true, while a monitor on the distribution would have shown that code jump from two per cent of rows to eleven overnight.

A data contract is where the promise is written down: schema, service levels for freshness and availability, quality rules with thresholds, a named owner, and the policy for breaking changes. The Open Data Contract Standard gives that promise a platform-agnostic shape. What makes a contract real is that it lives in version control and is checked on every pull request, so a change that would break a stated consumer fails before it merges.

Zhamak Dehghani set out data mesh in 2019 and 2020 as a sociotechnical pattern for scaling analytical data, arguing that one central team cannot hold every domain's meaning, quality and change cadence. Its four principles are commitments rather than labels: domain teams own the analytical data they produce, including its meaning and change control; those outputs run as products with documented consumers, service levels, versioning and a support route; a shared self-serve platform stops each team rebuilding infrastructure; and governance is federated, decided centrally and enforced through code, tests and platform controls.

The leadership test is short: ask who fixed the last three data incidents in a named domain. If the central team still fixes everything while domain teams attend workshops, the organisation has taken the vocabulary and not the operating model. Ownership with no budget, no on-call and no authority to refuse a request is a diagram.

This is also where contracts stop being a quality device and become the product's interface, stating what a consumer receives and what the producer guarantees. The post-hype reading is that mesh is worth adopting where domain ownership solves a scaling problem the organisation can actually name, and that a small team can take the product thinking, the ownership and the contract without taking the bureaucracy.