Resilience and Fault Tolerance

In 1994, Peter Deutsch at Sun Microsystems documented a set of assumptions that developers routinely make about distributed systems and that are reliably false. These became known as the eight fallacies of distributed computing. Understanding them is the starting point for resilience engineering, because the failure modes that produce real production incidents are almost always a consequence of one of these assumptions being violated.

The most consequential fallacies for system design are: the assumption that the network is reliable (packets are lost, connections drop, switches fail); the assumption that latency is zero (cross-data-centre calls add 10 to 100 milliseconds, and any service call can become arbitrarily slow under load); and the assumption that the network topology does not change (services restart, scale in and out, and IP addresses change in dynamic cloud environments).

A system designed on these assumptions behaves correctly in development and staging, where the network is local and reliable, and fails unpredictably in production. Resilience engineering means designing the system to continue functioning at a reduced level of service when these assumptions are violated, rather than failing completely.

When a downstream service becomes slow or unavailable, every request to it blocks a thread waiting for a response that may take 30 seconds to time out. If the calling service has 50 threads and 50 requests are blocked waiting for the failing service, no threads are available to handle any other requests, including requests that have nothing to do with the failing service. The entire calling service becomes unresponsive. This is cascading failure.

The circuit breaker pattern, coined by Michael Nygard in Release It!(2007) and popularised by Netflix Hystrix, addresses this by monitoring the failure rate of calls to a downstream service and opening the circuit when failures exceed a threshold.

The circuit has three states. In the closed state, calls pass through normally and the failure rate is tracked. When the failure rate exceeds the threshold (for example, five failures in ten seconds), the circuit opens. In the open state, calls fail immediately without attempting the network call, returning a cached fallback or an explicit error. After a configurable timeout, the circuit enters the half-open state, allowing a single test request through. If the test request succeeds, the circuit closes. If it fails, the circuit opens again.

The circuit breaker converts a slow failure (waiting 30 seconds for a timeout) into a fast failure (returning an error immediately). This frees threads to serve other requests and allows the calling service to return a degraded response rather than blocking entirely.

The bulkhead pattern takes its name from the compartmentalised hull sections of a ship. If one compartment floods, the bulkheads prevent water from reaching the others. In software, a bulkhead isolates failure by ensuring that a problem with one downstream dependency cannot exhaust the resources (threads, connections) used by all other operations.

The implementation is separate resource pools per downstream dependency. Rather than a single shared thread pool handling all outgoing calls, each downstream service gets its own pool. If the Payment service pool fills with blocked threads waiting for a slow response, the Account service pool is unaffected and continues to serve requests normally.

In practice, bulkheads are implemented using separate connection pools (limiting the number of concurrent HTTP connections to each downstream service independently), separate thread pools in thread-per-request models, or separate async worker pools in event loop models. The key parameter is the limit: each pool should be sized to the capacity of the downstream service under normal load, not to the maximum concurrency of the calling service.

Every network call must have an explicit timeout. A call without a timeout blocks indefinitely if the downstream service never responds. In a microservices system, an indefinitely blocking call propagates up the call chain: the Checkout service calls the Inventory service, which calls the Pricing service, which has no timeout on its database call. A slow database query blocks the entire call chain.

Timeouts should be set at each layer independently, and the outer timeout should be longer than the inner one to allow for retries. A user-facing API should timeout in 2 to 5 seconds; an internal service call should timeout in 500 milliseconds to 2 seconds; a database query should timeout in 100 to 500 milliseconds.

Retries are appropriate for transient failures: temporary network interruptions that resolve within milliseconds. Use exponential backoff with jitter, increasing the delay between each retry by a factor and adding random variation to prevent all retrying clients from hitting the downstream service simultaneously (the thundering herd problem). Do not retry non-idempotent operations without idempotency controls: retrying a payment creation request without an idempotency key charges the customer twice.

Graceful degradation means serving a reduced-quality response rather than failing completely. When the Recommendation service is unavailable, the product page shows popular items from cache instead of personalised recommendations. Define explicit fallback strategies for every external dependency before the dependency fails in production, not after.