Streaming and change data capture

A scheduled data pipeline that runs once a night is the cheapest reliable thing in data engineering. It has one start time, one finish time and one failure mode. If it breaks, an engineer reruns it in the morning and the world is restored. There is no state to recover, no ordering to preserve and no consumer waiting mid-flight. The operating cost is close to the compute cost.

A continuously running stream has none of those comforts. It is always up, so it is always able to fail, and failure arrives at three in the morning rather than during office hours. It holds state between records, so recovery means restoring that state as well as restarting the process. It has consumers that are part-way through the data, so a restart has to answer the question of where to resume. Every one of those properties is engineering work that the nightly job never had to do.

That is why the useful question is not whether streaming is better. It is which decisions change when the data is five seconds old instead of twelve hours old, and what the improved decision is worth. Card fraud scoring, grid balancing, dynamic pricing, fulfilment promises and safety alarms all change materially. Monthly management reporting, regulatory returns, cohort analysis and most machine learning training sets do not change at all. Buying seconds of latency for the second group is pure cost.

Once the answer is known, write it down as a number rather than an adjective. A freshness target expressed as a service level objectivegives the producing team a line they can be measured against and the consuming team a fact they can design around. "Near real time" is not a target. "The orders table is at most ninety seconds behind the source, measured at the ninety-fifth percentile over a rolling day" is a target, and it can be breached, alerted on and argued about with evidence.

An append-only log is a sequence of records that is only ever added to at the end and never edited in place. It is therefore an ordered history of what happened rather than a snapshot of what is currently true. Current state is derived by replaying the records from a known position. This is not a streaming invention: it is the same structure a relational database uses for durability, which is why the two worlds join up so neatly later in this module.

An event stream exposes that log to consumers. In Apache Kafka, the unit consumers subscribe to is a topic, and each topic is split into partitions held on different brokers. Ordering is guaranteed within a partition and nowhere else. That constraint is easy to read past, because from the outside a topic looks like one stream. If every event about one customer must be processed in order, the customer identifier has to be the partition key, and the number of partitions becomes a design decision rather than a capacity setting.

A consumer group is how a set of processes share the work of reading a topic. Each partition is assigned to exactly one consumer in the group at a time, so the maximum useful parallelism of a group equals the partition count. Adding a tenth consumer to a six-partition topic buys nothing. Each group tracks its own position, called an offset, which is why several unrelated groups can read the same topic independently: the fraud engine, the warehouse loader and the alerting service each move through the same events at their own pace without interfering.

Events are retained for a configured period rather than deleted when they are read. Retention is what makes a stream recoverable. A consumer that has been writing corrupted output for six hours can be fixed, reset to an earlier offset and replayed, which is impossible with a queue that discards a message once it has been delivered. The retention window is therefore a governance decision as much as a storage one: it sets how far back a mistake can be undone, and it sets how long personal data sits in the platform.

For most of its life a Kafka cluster was two distributed systems, not one. Kafka held the data and Apache ZooKeeper held the cluster metadata: which brokers exist, which partitions live where, which configuration applies. Operators had to secure, patch, monitor and capacity-plan both, and a ZooKeeper problem became a Kafka problem.

Apache Kafka 4.0, released on 18 March 2025, was the first major release to operate entirely without ZooKeeper. KRaft mode, in which controller nodes inside Kafka manage cluster metadata as an internal log, became the default. The technical change is modest and the operational change is not: one distributed system to run instead of two, one security model, one upgrade path and one set of alerts.

The 4.0 release also introduced share groups, described in KIP-932 as queues for Kafka, as an early access feature. A share group behaves roughly like a durable shared subscription: records from a partition can be handed to several consumers cooperatively, so consumer count is no longer capped by partition count for work-queue style processing. The announcement marks the feature as early access with documented limitations, which is the correct way to read it. It is a signal about where the platform is going, not a foundation to place a production settlement pipeline on this year.

For a course on data rather than on Kafka administration, the point to carry forward is the direction of travel. Streaming infrastructure is consolidating: fewer moving parts, fewer specialist operators, and messaging patterns that used to require a separate broker now being absorbed into the log. That lowers the cost of the infrastructure and shifts the difficulty upwards, into modelling, semantics and delivery guarantees, which is exactly where the remaining sections sit.

The traditional way to find out what changed in a source system is to ask it repeatedly. A job selects every row whose modified timestamp is later than the last run and loads the result. This is simple, it works, and it quietly loses data in four separate ways.

First, it depends on a modified timestamp that every writer maintains correctly, which is rarely true across an estate that includes batch scripts and vendor packages. Second, it cannot see deletes: a row that has gone leaves nothing behind for a query to find, so downstream copies keep records the source has removed. Third, it misses intermediate states. If a row changes three times between polls, the poll sees only the final value, and an audit trail built this way is fiction. Fourth, every poll is analytical load applied to an OLTP system that was sized for short transactions, so the freshness you gain by polling more often is paid for by the customers waiting on the application.

Change data capture takes a different route. Relational databases already write every committed change to a durable log before applying it to the data files, because that is how they survive a crash. That log is a complete, ordered record of inserts, updates and deletes that the database maintains whether anyone reads it or not. Change data capture reads it and republishes each row-level change as an event.

Debezium is the open-source implementation most teams meet first. Its connectors read the source database log and emit an event per change, and only committed changes are published, so consumers never have to reason about work that was later rolled back. A connector normally begins with a consistent snapshot of the existing tables, then switches to reading the log from the position that snapshot corresponds to, so the stream is continuous from the first event. Deletes appear explicitly, which is what makes downstream retention and erasure work possible at all.

Three obligations come with the approach. Schema changes in the source now propagate into the stream, so someone has to decide what happens to consumers when a column is renamed. Personal data flows out of the operational estate into the platform, so the lawful basis and retention rules travel with it. And the managed zero-ETL services that cloud vendors offer remove the pipeline you would otherwise operate, but they do not remove the modelling, the quality rules or the contract. Fast access to badly understood tables is not progress.

Networks drop packets, processes are killed mid-write, and brokers fail over. Every streaming design therefore has to state what happens to a record when the path between producer and sink is interrupted. There are three possible answers, and they differ in what they are willing to lose.

At-most-once means the producer does not retry. If an acknowledgement is lost, the record is gone. Nothing is ever duplicated and some things are simply missing. This is acceptable for high-volume telemetry where a gap changes no decision, and it is indefensible for anything financial.

At-least-once means the producer retries until it is sure the record landed. Nothing is lost and some records arrive twice, because a retry after a lost acknowledgement writes the same record again. This is the default behaviour of most systems and the right starting assumption when reading someone else's pipeline.

Exactly-once delivery means each record affects the final result once, even though producers retry and consumers restart. It is assembled rather than switched on: producers must be able to detect and discard their own duplicate writes, the processing step must commit its state and its output together, and the sink must be able to participate in that commit. Kafka documents these three semantics directly and explains that its exactly-once processing rests on duplicate-suppressing producers and transactional writes.

The practical consequence is that exactly-once processing is a property of the whole path, not of one component. A pipeline that enables transactional writes in the stream processor and then appends to a sink that cannot commit transactionally has not achieved it, whatever the configuration file says. The honest test is to walk every hop from source to final table and ask whether that hop can keep the promise.

There is a cheaper route that covers most real requirements. Run at-least-once and make the final write idempotent: upsert on a natural key rather than appending, so a duplicated record overwrites itself instead of double-counting. The effect at the point of use is exactly once, and the cost is a key and an upsert rather than a distributed transaction. Reserve the full guarantee for payments, billing and settlement, where a duplicate is a real financial event, and decline to pay for it on clickstream.

Kafka moves events. It does not compute over them. That job belongs to a stream processing engine such as Apache Flink, which treats the input as an unbounded sequence rather than a finished table, holds state between records, and produces results continuously. The arithmetic in these jobs is usually trivial. The engineering cost is the state: what has to be remembered, for how long, and how it is restored after a failure.

Time is the second cost. A streaming job orders work by the time events happened rather than the time they turned up, because a mobile device that was offline for an hour will deliver an hour-old event now. A watermark is the marker carried along the stream that declares how far event time has advanced, which is how the job decides that a window is complete and emits its result. Flink documents a watermark at time t as the declaration that no further elements with a timestamp at or before t should arrive.

That definition contains the trade-off every streaming design has to face and state openly. Wait longer before closing a window and every result is later but more of the stragglers are included. Wait less and results are prompt but some are wrong. There is no configuration that removes the choice, and a job that has no stated policy for late data has made the choice by accident.

The output has to land somewhere. The streaming lakehouse pattern writes it into the same governed tables that batch jobs write and query engines read, using an open table format such as Apache Iceberg to give files in object storage the behaviour of tables: atomic commits, schema changes that do not rewrite history, and queries against an earlier snapshot. That removes the old split in which streaming results lived in one store and batch results in another, and the two disagreed.

Two costs come with it. Streaming writes produce many small files, so compaction and snapshot expiry become scheduled operational work rather than optional tuning. And a table written by several engines at once makes the table format the interoperability contract for the whole platform, which is a strategic commitment rather than a storage detail.