Scalability Patterns

Vertical scaling (scaling up) means adding capacity to a single server: more CPU cores, more RAM, faster storage. It is operationally simple: the application requires no changes and deployment complexity does not increase. The limits are hardware maximums, which are finite and increasingly expensive at the high end, and a single point of failure: if the server fails, the service is unavailable.

Horizontal scaling (scaling out) means adding more instances of a service and distributing load across them via a load balancer. It has no theoretical upper limit: add more instances to add more capacity. Each instance can fail without taking the entire service down. The cost scales linearly with instance count rather than superlinearly as with high-spec servers.

Horizontal scaling requires that application servers be stateless. If a server stores session state locally (a user's login session in memory), a second server cannot serve that user's subsequent requests. All state must be externalised: session state in Redis, user data in the database, uploaded files in object storage. Any instance must be able to serve any request.

Caching stores the result of an expensive computation or database query so it can be returned quickly on subsequent requests without repeating the work. It is the most impactful single optimisation for read-heavy systems, and also one of the most common sources of subtle bugs.

The cache-aside pattern is the most common implementation: the application checks the cache first; on a cache miss, it fetches from the database, stores the result in the cache with a time-to-live (TTL), and returns it to the caller. On the next request for the same key, the cache returns the stored value directly.

Caching can be applied at multiple layers, each with different trade-offs. A CDN (Content Delivery Network) caches static assets and HTML pages at edge locations geographically close to users, reducing both latency and origin server load. An API gateway can cache GET responses for public, idempotent endpoints. An application cache (Redis is the standard) stores database query results and computed values with configurable TTLs.

The invalidation strategy must be designed before implementing the cache, not after discovering stale data in production. TTL expiry is the simplest approach: the cache entry expires after N seconds. Write-through invalidation updates the cache on every write, keeping it consistent at the cost of write overhead. Event-driven invalidation uses domain events to explicitly remove or update cache entries when the underlying data changes, providing stronger consistency at higher implementation complexity.

Application servers scale horizontally with relative ease: they are stateless, and adding a new instance requires only provisioning and registration with the load balancer. Databases are harder because they hold shared, mutable state. Two strategies address the most common database bottlenecks.

Read replicas address read throughput. The primary database handles all writes. One or more read-only replicas receive a copy of all writes via replication and serve read queries. The majority of database operations in most applications are reads; routing them to replicas reduces load on the primary dramatically. The cost is replication lag: replicas are slightly behind the primary. For most reads (a product catalogue, a news feed), this is acceptable. For reads that must reflect a just-completed write (a user viewing their own updated profile), the application must read from the primary.

Sharding addresses write throughput and data volume. The dataset is partitioned horizontally across multiple database instances. Each shard holds a subset of the data. Queries for a given record go to the shard that holds it. Sharding allows the write load to be distributed across multiple primaries, breaking through the vertical scaling limit of a single database.

The cost of sharding is significant: cross-shard queries are expensive and sometimes impossible without application-level joins; re-sharding when the partition key produces uneven distribution requires data migration; transactions across shards require distributed transaction protocols. Sharding should be adopted only when simpler strategies (vertical scaling, read replicas, caching) have been exhausted and a specific write throughput or data volume bottleneck has been identified.

Queues decouple the rate at which work is accepted from the rate at which it is processed. This is valuable for absorbing traffic spikes: during a peak, the API enqueues requests at the incoming rate (which can be very high) and workers process them at a sustainable rate (which is bounded by their capacity). The queue absorbs the difference.

Without a queue, a traffic spike that doubles the incoming request rate requires either doubling the processing capacity (which may not be possible instantly) or dropping requests. With a queue, the spike is absorbed and processed as capacity allows, at the cost of increased latency for requests submitted during the peak.

The queue depth is the key operational metric: a growing queue means workers are not keeping up. Auto-scaling workers based on queue depth is the standard cloud pattern. Amazon SQS with Lambda or ECS auto-scaling, and Google Cloud Pub/Sub with Cloud Run, both support this pattern natively. The queue depth threshold for scaling triggers should be set based on the acceptable latency SLO (Service Level Objective) for the queued work.

Queue-based load levelling is appropriate when the work is asynchronous and the caller does not need an immediate result: image processing, report generation, email delivery, and data exports are all good candidates. It is not appropriate when the user must wait for the result in their current session, which requires a synchronous response or a polling mechanism.