CDNs, anycast and the edge

Start with the one quantity in networking that no budget can buy away. A request has to travel to wherever the answer is and the answer has to travel back. That round trip has a floor set by the distance and the medium, and the floor does not care how many processors the destination has, how well the code is written or how much the hosting costs. Buy a faster server and you shorten the part of the wait the server owns. The journey is untouched.

Now recall what a modern connection actually costs in round trips, because the foundations stage measured it. Resolving the name is a journey. Opening the transport connection is a journey. Agreeing keys is a journey, or part of one. Sending the request and waiting for the first byte of the reply is another. Those journeys multiply the distance rather than adding to it, which is why a service that feels instant to the team that built it, sitting in the same city as the server, feels sluggish to a user on another continent running identical code against identical hardware. The difference between those two experiences is geography.

Once the problem is stated that way the answer is forced. If you cannot make the journey faster, make it shorter, which means putting the answer near the person asking. That is the entire idea of a content delivery network: a set of caches placed close to users that serve content on the origin's behalf, together with the request routing that decides which cache a given user reaches. The origin stays authoritative and keeps the truth. The caches carry the traffic. Each of the sites holding those caches is a point of presence, and where those sites sit is usually the largest single lever anyone has over time to first byte, because it is the only lever that operates on distance rather than on work.

Which raises the question the rest of the section answers, and it is a harder question than it looks. How does a browser in Lisbon end up talking to a machine in Lisbon, given that the user typed a name, the name resolved to an address, and neither the user nor the browser knows that any of this exists?

The obvious answer is to hand out different addresses to different people, and it is not the answer content networks rely on. The mechanism they rely on is anycast: announce the same address from many separate locations and let ordinary routing deliver each client to one of them. The client does nothing special and is told nothing. It sends to one address. The routing system, doing exactly what it does for every other packet, delivers to whichever announcement of that address it considers best from where the packet entered.

Three consequences follow, and they are the ones worth carrying out of this section. The first is that nearest means nearest in routing terms and not in miles. Routing chooses by the path metrics and business relationships that the routing module took apart, so a user can be served by a site that is further away on a map and closer along the wire, and occasionally by one that is neither, because somebody's routing policy said so. The second is that the choice is invisible from both ends: the user is never told which site answered, and the operator of the origin cannot see the decision being made either. The third is that this is not steering that anybody performs. There is no lookup table of users and no geographic database being consulted per request. Routing is doing what it already does, and the edge selection is a by-product of it.

Naming the machine that answers is only the beginning of the argument. The reason the edge matters to an engineer is not that it is nearby. It is what it does once the request lands there, starting with the fact that the encrypted connection your user believes they have with you actually ends at a machine you do not own.

When the request lands at the edge, the first thing that happens is the thing people find hardest to accept. Transport Layer Securityterminates there. The certificate the browser checked, which carries the site's name and satisfied every validation rule the TLS module described, was presented by the edge. The keys were agreed with the edge. The encrypted channel the user trusts runs from their device to a machine in a rented rack a short distance away, and it stops there.

Nothing about that is a defect, and TLS is not being weakened by it. The protocol protects a connection between two endpoints, and the edge is genuinely one of those endpoints. What has moved is who the far endpoint belongs to. A second connection, which the user never sees and which usually carries its own separate encryption, runs from the edge back to the origin when the edge needs something. Two connections, two handshakes, two sets of keys, one address bar.

Follow that through and the argument of this section arrives on its own. If the connection ends at the edge, then the edge is what parses the request. It decides which HTTP version the conversation uses. It applies compression, rewrites or adds headers, enforces the rate limit, decides what a request with no matching route deserves, and decides how long it is prepared to wait for the origin before giving up and producing an error page of its own design. Users experience all of that as the behaviour of your server. Most of it is the behaviour of an intermediary you rent, and a good deal of it is governed by a configuration held in that vendor's dashboard rather than anywhere in your repository.

That is the practical reason engineers get stuck on incidents in this stage of the course. A user reports an error your application cannot produce, carrying a reference number that appears in none of your logs, at a moment when your service was healthy. All three observations are true at once, and they are consistent, because the thing that answered was not your service.

The figure above stacks the four places an answer can come from, and it is worth reading it as a sequence of questions rather than as a diagram of boxes. The first question is asked on the device itself. The browser cache holds responses this one user already fetched, kept for as long as the response headers allowed, and a hit there means no request leaves the machine at all. This is the cheapest cache in the system and the one operators think about least, which is why response headers that nobody has reviewed for two years are such a common finding.

A miss sends the request to the edge cache at the point of presence that anycast selected. That cache holds objects other nearby users have already pulled through the same site, which is the first point at which one person's request pays for another person's speed. A hit here is the route the primary figure calls route A: the reply travels one short hop back and nothing else in the world moves.

A miss at the edge is where the design gets interesting, and where a naive content network hurts the origin it was bought to protect. Consider what happens without the next layer. A popular object expires. Every point of presence discovers this independently, at roughly the same moment, because they were all serving the same object to their own local users. Every one of them then asks the origin. One expiry produces one request per site, all in the same second, at exactly the moment the object is most in demand. The origin sees a spike that looks like an attack and is in fact its own cache doing its job.

An origin shieldremoves that. It is a caching tier placed between the many edges and the customer's origin, and the rule that makes it work is a restriction rather than an addition: an edge that misses must ask the shield, and only the shield is allowed to ask the origin. The many simultaneous misses collapse into one fetch. The rest are served from the shield, which is further from the user than the edge and much closer than the origin, and which exists precisely so that the edges do not each ask.

Only a miss at the shield reaches the origin, the one layer in the stack that is not a cache because it builds the response rather than storing somebody else's. And on the way back the reply fills every layer it passes through, which is the sentence that explains cache behaviour better than any hit-rate figure. A copy exists only because an earlier request missed and filled it. The first user through pays for the fill; everybody behind them does not.

Now put the two halves of this section together, because the combination is what changes an engineer's working practice. The connection ends at the edge, and most requests never travel past it. Therefore the request log at your origin is not a record of user demand. It is a record of cache misses, which is a small and heavily biased sample: it over-represents the unusual, the uncacheable and the first request after every expiry, and it under-represents everything popular enough to matter. Reading origin traffic as if it were user traffic will mislead you about load, about which pages are used, about which countries you serve and about when your busy period actually is.

Everything so far has assumed the client reaches the edge in the first place. Anycast gets it there once it has an address to send to, and getting an address is the job of a system this course has already taught. What has changed since that module is how much more than an address the client now asks for, and how much of the connection is decided before a single packet is sent to the destination.

The foundations module on the path from a URL to the first byte ended with a correction to the mental model most people carry: the browser no longer looks up an address and then connects. It asks DNS how to connect, and then races. This section is where that correction pays off, because the record it asks for is the mechanism that lets a content network steer a connection before the connection exists.

An address record answers one question and only one: which address. That was sufficient when every service on the internet spoke the same protocol on the same port. It stopped being sufficient once an endpoint might speak several HTTP versions, might prefer a different name for the actual service, might want a different port, and might hold keys that the client needs before it can send the first encrypted message. A client with only an address has to guess, connect, discover it guessed wrong and try again, and every one of those steps is a journey with the floor described in section 28.1.

The HTTPS record is the form of that record specific to the http and https schemes, and the parameter that matters most for this module is ALPN. Listing the protocols an endpoint speaks in DNS means a client can start on HTTP/3 from its very first connection instead of discovering later that it was available.

Compare that with how the same discovery worked before. HTTP/3 announces itself during the handshake through the ALPN token h3, so the client learns the endpoint speaks it at the moment it is already connecting. To know in advance, a browser had to have been told on a previous response, through the Alt-Svc header field, that the same resources were also available over another protocol. That works, and it has one structural weakness: it requires an earlier connection over the older protocol. The first visit always pays. Moving the announcement into DNS moves it before the connection, so the first visit stops paying, and a service can shift its clients between endpoints and protocols by publishing a record.

It is worth being precise about the division of labour between the two steering mechanisms in this module, because they are frequently confused and they fail in opposite ways. DNS steering chooses which endpoint, which protocol and which parameters, and it is coarse: the answer is cached for as long as its lifetime allows, it is the same answer for everyone who asked that recursive resolver, and the authoritative server that produced it saw the resolver rather than the user. Anycast chooses which instance of the endpoint, and it is fine-grained and current: it is decided by routing, per packet, from wherever the packet actually entered the network, with no cache and no guess about where anybody is.

That is why serious content networks use both rather than picking one. DNS carries the part that has to be known before the connection and changes slowly. Routing carries the part that has to be right at the moment of the packet. A design that tries to do proximity selection in DNS alone inherits every caching and vantage-point problem in the resolution chain, and this course has already shown how long a stale answer can outlive the thing it described.

One further consequence belongs to the security stage rather than to performance. If the connection instructions live in DNS, then whoever answers DNS for your users has acquired influence over which endpoint they reach. Encrypted resolution moved that influence from the local network to the resolver the client chose, which the DNS module treated as a genuine trade rather than as a straightforward improvement. The same record type also carries the keys for encrypted client hello, which closes the gap where the requested name travelled in clear text at the start of the handshake, and closes it only for clients that fetched the record.

So far every capability in this module has been about moving copies of an answer closer to people. The next step is the one vendors sell hardest, and it is a different claim: not moving the answer, but moving the code that produces it.

The same estate of points of presence that stores objects can also run code, and the platforms that do it are familiar: Workers and Lambda@Edge class services, where a short piece of application logic executes at the site that answered rather than in one central region. That is edge compute, and the honest way to place it is with one sentence you should keep. It buys latency, not capacity.

Test any proposal against that sentence and most of the argument resolves itself. Work that is round-trip bound gets faster, because the round trip is what shortened. Work that is bound by anything else does not, because nothing else changed. The processor at the edge is not faster than the processor in your region. The database is not closer. The dataset your code needs did not move. All that moved is the place the code runs, so only the delays that were caused by distance can shrink.

That gives a usable rule for what belongs there: work that decides something about the request itself, using only what the request carries, and where the decision is what the user is waiting for. Redirects and rewrites qualify. Choosing which variant of a page a visitor sees qualifies. Validating a signed token before anything expensive is started qualifies, and it is the case the zero-trust module argued for, because a request that should be refused is best refused before it crosses an ocean. Setting or reading a cookie, normalising headers, blocking an obviously abusive client, serving a maintenance page while the origin is unreachable: all of these are decisions the request already contains the inputs for.

The rule excludes more than it admits, and the exclusions are where money is wasted. Anything that has to read or write a single authoritative copy of state does not belong at the edge, because the write has to reach whichever region holds that copy and you have simply moved the code away from the data it needs. Anything working over a large dataset does not belong, for the same reason. Anything with a data-residency obligation is worse than slow, because code that runs in hundreds of locations is code that processes data in hundreds of jurisdictions, and that is a legal question rather than an engineering one.

The worst case is the one that looks like a success in the demonstration. Move a piece of application logic to the edge, keep the database in one region, and every request now travels from the user to a nearby site, from that site to the distant region and back, then back to the user. The distance was not removed. It was paid twice, with an extra hop added, and the profile of the request looks better in exactly one metric, which is the time to first byte of a response whose useful content arrives later.

Two operational facts round the section out, and both follow from the same property that makes edge compute attractive. Code that runs in hundreds of places fails in hundreds of places independently, so an incident is normally partial: it affects the users routed to a subset of sites and looks like nothing at all from anywhere else, which is precisely the shape of failure the practice capstone is built around. And a deployment reaches those places over a short but real interval, so two users can be running different versions of your logic at the same moment. Neither fact is a reason to avoid the technique. Both are reasons to instrument it per location and to design for a request that might meet either version.

Every proxy in this module so far has been one you chose, paid for and configured. The last section is about the same machinery arriving from the other direction, put there by the user, and it is the part that changes what your logs mean.

Proxies are not new and this course has met several. What is new is where they live. The proxies of the previous decade sat at a boundary somebody administered: the corporate web gateway, the reverse proxy in front of an application, the transparent cache an internet provider installed. They were infrastructure, imposed on the traffic that happened to pass through them.

MASQUE inverts that. It is proxying expressed as an ordinary request inside an ordinary HTTP connection, made by the client, to a server that agrees to act as a proxy. RFC 9298 covers the UDP case, known as CONNECT-UDP: the client asks the proxy to open a UDP association on its behalf and then exchanges HTTP datagrams with it, which is what carries QUIC and therefore what carries most modern web traffic. RFC 9484, CONNECT-IP, does the same at the IP layer, so an entire packet flow rather than a single association can be tunnelled the same way.

Notice what that means for anyone watching the connection. The tunnel runs inside a normal HTTPS conversation, to a normal HTTPS port, over the same transport as the rest of the web. There is no distinctive protocol on a distinctive port to recognise. A network that wishes to identify this traffic is reduced to identifying the proxy operator's addresses, which is a policy decision about named companies rather than a protocol control, and one that ages badly.

The story at the top of this module is the deployed example, and its two-hop structure is the part worth understanding rather than the branding. One relay knows who the user is and not where they are going. The other knows the destination and not who asked. Neither can produce the record that a single proxy would hold by default. The design is a deliberate separation of two facts that are only sensitive when combined, and it is enforced by architecture rather than by policy, which is the same argument the zero-trust module made about not trusting a component more than the design requires.

For the operator of a service, all of this arrives as one blunt change: the client address you record may belong to a relay. Work through what that breaks, because each item is a control that a lot of estates still rely on.

Geolocation by address stops describing the user and starts describing the relay egress, which may be in a different country to the person. Blocking or allowing by address stops being a statement about one network and becomes a statement about everyone sharing that egress, which is the classic way a well-intentioned block takes out a population of ordinary customers. Rate limiting by address counts a crowd as one client, so the limit either lets abuse through or throttles legitimate users, and it is not obvious from the metrics which of those you have chosen. Treating an address as a stable identifier for one person, across a session or across days, stops working because the egress can change and because two unrelated users can share one. And any fraud or abuse score that weights an address is now weighting a property of a proxy service that the user did not choose per request.

The remedy is not to detect and defeat the relay. It is to stop asking the address for information it never reliably held. Identity comes from authentication, and this course has already spent a module on why the authenticated principal, evaluated per request, is the control that survives contact with reality. Rate limits attach to accounts, sessions or tokens. Location, where you genuinely need it, comes from something the user tells you or from a business record such as a delivery or billing address, both of which are honest about being claims. The address stays useful for what it is: a network-layer fact, good for diagnosing a path, not for identifying a person.

One caveat keeps this from becoming a counsel of despair. Relayed traffic is still ordinary traffic once it arrives. Your edge still terminates TLS with the client, the request still carries whatever the application put in it, and the session is still the session. What you lost is a proxy for identity that was always weak and is now obviously so, which is the same lesson the IPv6 module reached from a completely different direction when privacy addressing made a laptop show several addresses at once.

There is a measurement point hiding in all of this, and it closes the loop with a discipline this course keeps returning to. Cloudflare Radar, which this course uses as a public window on protocol adoption, publishes what it publishes because it operates one of these edges and can see the traffic that arrives there. Measurement literacy says to name that vantage point rather than ignore it: the figure describes the population that reaches that particular network, from wherever anycast routed them, seen at the layer the edge terminates. That is a real measurement, published with its method, and it is not the same thing as a measurement of the internet. The whole of this module is a reason why: what an edge can see is exactly what did not have to travel any further.

The check below runs the module's central distinction as it appears in practice, which is somebody asking where a request ended when the answer decides who has the evidence.