Event-Driven Architecture

In a request-driven system, service A calls service B and waits for a response. The caller and the callee are coupled: A must know B exists, know B's API, and be affected by B's availability. If B is slow, A waits. If B is down, A fails.

In an event-driven system, service A publishes a fact: “an order was placed.” It does not call service B. It does not know that service B exists. Service B subscribes to order-placed events and reacts when they arrive. If B is slow, A is unaffected. If B is down, events queue until B recovers.

This difference in coupling model has a compound effect on large systems. LinkedIn's analytics, search indexing, and feed ranking systems all consume the same activity event stream from the same Kafka topic. Adding a fraud detection service in 2022 required zero changes to any producer. The producer published; the new consumer subscribed. That is loose coupling in practice.

Three message types flow in distributed systems. Distinguishing them prevents architectural confusion.

Domain events record facts within a bounded context:OrderPlaced, StockLevelChanged. They are past-tense and owned by the producing service. Consumers react; producers do not instruct.

Integration events translate domain events for consumption by external bounded contexts. The Order Service's OrderPlaceddomain event becomes an OrderConfirmedForShipping integration event for the Logistics bounded context. This translation layer prevents the logistics system's language from bleeding into the order domain.

Commands are instructions sent to a specific receiver:ProcessPayment, SendEmail. They are imperative and directed. If the receiver does not exist or refuses, the command fails. Commands are appropriate for request-driven interactions where the result matters.

The distinction is not cosmetic. An event-driven system built with commands disguised as events still has the coupling of the command model. True events require producers to be unaware of consumers.

Apache Kafka uses a log-based model. Events are appended to a partitioned, replicated commit log. Each partition is ordered. Consumer groups track their own position (offset) in the log and read at their own pace. Events are retained for a configurable period, from hours to years. Multiple independent consumer groups can read the same events simultaneously, each from their own offset. This enables replay: a new service can read the entire event history on first deployment.

RabbitMQ uses a queue-based model. Messages are delivered to queues and deleted after acknowledgement. The broker manages delivery state. Consumers do not track offsets. RabbitMQ routes messages through exchanges with flexible routing rules: direct, topic, fanout, and headers exchanges. It is operationally simpler than Kafka for task distribution workloads where each message must be processed exactly once by one worker.

AWS EventBridge is a serverless event bus. It receives events from AWS services, custom applications, and SaaS partners, routes them via rules to targets (Lambda, SQS, SNS, Step Functions). It is suited to event routing in cloud-native AWS architectures where managed infrastructure is preferred over operating a Kafka cluster.

Choose Kafka for high-throughput streams that require replay, multiple consumer groups, and long-term retention. Choose RabbitMQ for task queues and work distribution. Choose EventBridge for AWS-native event routing without operational overhead.

Kafka guarantees ordering within a partition, not across partitions. Events for the same order should be published to the same partition to preserve their ordering. Partition assignment is controlled by the message key: Kafka hashes the key to determine the partition. Using the order ID as the key ensures all events for order-1042 land in the same partition and arrive in order.

Kafka's default delivery guarantee is at-least-once: in the event of producer retry or consumer reprocessing, the same event may be delivered more than once. This means consumers must be idempotent: processing the same event twice produces the same result as processing it once. A consumer that inserts a payment record on each delivery will create duplicate payments; a consumer that upserts using the event ID as the unique key is idempotent.

Exactly-once semantics (EOS) in Kafka requires both producer idempotence (enabled via enable.idempotence=true) and transactional consumers. EOS is supported in Kafka 0.11+ but adds latency and operational complexity. Most production systems use at-least-once delivery with idempotent consumers rather than exactly-once, accepting the operational simplicity trade-off.

Event schemas must evolve without breaking consumers. The Confluent Schema Registry with Apache Avro is the standard solution. Producers register schemas with the registry; consumers validate received events against stored schemas.

Backward compatibility means a new schema can read data written with an old schema. Adding a field with a default value is backward-compatible: old consumers see the new field with its default and continue working.

Forward compatibility means an old schema can read data written with a new schema. Removing a field is forward-compatible if the consumer ignores unknown fields: old consumers simply never receive the removed field.

Breaking changes are changes that neither old nor new consumers can handle: renaming a field, changing a field's data type, or removing a required field. Breaking changes require a schema version bump and a consumer migration strategy. In long-running Kafka topics with months of history, schema evolution discipline is not optional.