VPCs, overlays and Kubernetes networking

A mechanism swapped underneath a name that never moved is not a Cilium peculiarity. It is how the whole of cloud networking has been built, and it is why so little of it is new. The ideas are the ones module 23 finished with: a boundary that decides which paths exist, and a stateful policy that decides what may cross. What moved is where those two things live and who is allowed to change them.

A virtual private cloud (VPC) is the boundary. It is a logically isolated network you define inside a cloud provider, with your own address ranges, subnets, route tables and gateways. Nothing reaches it from outside until you attach a gateway and write both a route and a rule that permit it. That last sentence is worth reading twice, because it names the two gates a packet has to pass and they are not the same gate. Routing answers whether a path exists at all. Policy answers whether traffic on that path is permitted. A cloud outage caused by a missing route looks nothing like one caused by a missing rule, and confusing the two is the single most common way an hour disappears.

A security group is the policy. It is a stateful allow-list attached to a resource saying which traffic may reach it and which may leave it. Because it is stateful, the reply to traffic you allowed out is let back in without a matching inbound rule, which is the same state tablebehaviour you met in the firewall and NAT material. The difference from a traditional design is placement. On premises, the enforcement point sits at a gateway in the middle of the network, so two machines on the same subnet talk without any policy seeing them. In a VPC, the policy is on the workload's own virtual interface, so it applies whether the other party is on the far side of the internet or on the next hypervisor.

That placement is why east-west traffic tends to be controlled in cloud estates almost as a side effect, where on premises it took a deliberate microsegmentation project with a budget. The knob simply sits somewhere more useful. It is worth being precise rather than triumphal about this: the default is better, the discipline still has to be applied, and a security group that permits everything from the whole VPC CIDR range recreates the flat network the design was supposed to prevent.

Three things did genuinely change, and they are the three that catch experienced network engineers. First, the control point is an API object, so a network change is a code review and a deployment rather than a console session, and it can be made by a pipeline at three in the morning. Second, because it is a template, a mistake replicates: the blast radiusof a bad rule is now the blast radius of whatever deploys that template. Third, and most consequential for the rest of this module, the address stopped being a stable identifier. Instances are created and destroyed constantly and addresses are recycled, so a rule written against an address range is a rule written against a moving target. The provider's own answer is to let one security group name another as its source, so the rule says which workloads may talk rather than which numbers may talk.

Naming workloads instead of numbers is the first appearance of an idea that section 24.5 finishes. Before that, there is a more basic question the VPC does not answer: if two tenants both use the same private address range, and both run on the same physical fabric, how does the fabric keep them apart without ever renumbering either of them?

The answer is the envelope in the figure above. The shared physical network is called the underlay, or the fabric: the same machines and links everything else runs on top of. The tenant network is drawn on top of it by wrapping each tenant frame inside a UDP packet addressed between two fabric nodes. The tenant's own addresses are never used for delivery across the fabric, so two tenants may use identical ranges and neither ever sees the other.

Read the four layers from the outside in, because that is the order a receiver strips them. The outermost is an ordinary IP packet whose source and destination are the two fabric nodes, not the workloads, and that is the only header the shared fabric reads. Inside it is a UDP header whose destination port tells the receiving node which encapsulation this is. Inside that is the overlay header carrying the tenant network identifier, which is what keeps one tenant's traffic from being delivered into another's. Innermost is the tenant frame itself, exactly as the workload sent it, addresses untouched.

This is encapsulationagain, the same mechanism the foundations stage introduced, applied one level further out. There, a web request became a TCP segment inside an IP packet inside a link layer frame. Here, a complete tenant frame becomes the payload of somebody else's UDP packet. The principle that each layer only reads its own header is what makes the whole arrangement work, and it is also what makes the fabric blind to the tenant on purpose.

Two formats do this job. VXLAN carries Ethernet frames inside UDP so a layer 2 segment can stretch across a routed layer 3 network, with a network identifier in the header keeping segments apart. Geneve performs the same wrapping but its header carries a variable-length options field, so the systems building the overlay can attach their own typed metadata to each tunnelled packet. That is the practical difference and the reason for the choice: if a platform needs to carry more than a tenant identifier from one end of the tunnel to the other, a fixed header leaves it nowhere to put that information.

What the envelope buys is worth stating plainly. Tenants may reuse address ranges. A workload can move to a different machine without renumbering, because its address lives in the inner frame and only the outer addresses change. And the fabric only has to know how to route between nodes, which is a far smaller and more stable problem than knowing about every workload.

What it costs is equally plain, and the first cost is the one that generates support tickets. Every packet now carries headers the tenant never sees, so the usable payload inside the envelope is smaller than the payload the underlay could carry. If the tenant's MTU is left equal to the underlay MTU, the first genuinely large packet either has to be fragmented or is dropped, and the symptom is the classic asymmetric one: small requests work perfectly, a large response hangs, and the connection looks established the whole time. The fix is to choose a tenant MTU below the underlay MTU deliberately, rather than leaving path MTU discovery to work it out in production against a network that may be silently discarding the messages it needs.

The second cost is evidential. A capture taken on the fabric shows outer addresses and an outer port, with the tenant's addresses sitting inside as payload. A filter written against the tenant addresses matches nothing at all on the fabric, which reads as "no traffic" when the truth is "wrong layer". That is the same lesson encapsulation taught in the foundations stage, arriving with real consequences: saying a packet was dropped means little until you say which layer you were watching.

An envelope explains how tenant traffic crosses shared fabric without collisions. It says nothing about how a workload gets an address in the first place, or how one workload finds another when both are replaced several times a day. Kubernetes answers both, and its answer pushes the interesting decision down into the datapath.

Kubernetes networking rests on three rules, and the whole model follows from them. Every pod, meaning the container or small group of containers that Kubernetes schedules and addresses as one unit, gets its own address. Any pod can reach any pod without address translation in the way, unless a policy stops it. And a group of interchangeable pods is fronted by one stable name and address so that callers never have to track individual pods.

The first rule is delivered by a plugin, not by Kubernetes itself. The Container Network Interface (CNI) is the specification and plugin model that decides how a container gets a network interface, and how that interface is cleaned up when the container goes away. Kubernetes hands pod networking to whichever CNI plugin the cluster runs and does not implement it. The consequence is the one that matters when you inherit a cluster: two clusters can look identical through the Kubernetes API and behave very differently on the wire, because the behaviour lives in the plugin.

This is also where section 24.2 rejoins the story. Some CNI plugins build a VXLAN or Geneve overlay between nodes, so pod traffic travels in envelopes and the MTU conversation above applies directly. Others route pod addresses natively in the underlay where the cloud provider allows it, so there is no envelope, no MTU deduction, and a capture on the node shows pod addresses in the clear. Same API, different wire. "Which CNI, and does it encapsulate?" is therefore the first question to ask about a cluster you have not seen before, and it is answerable in about a minute.

The second and third rules are what a Kubernetes Service exists for. Pods are disposable and their addresses change constantly. A Service gives a set of interchangeable pods one stable address and name, and spreads traffic across whichever pods currently back it. For the caller, that is the end of the problem: it connects to a name that does not move.

Here is the honest part, which most introductions skip. The Service address is virtual. No interface answers for it. Nothing owns it. It is a promise that something in the datapath has to keep, by turning that virtual address into a real pod address on every single connection. Which component does that, and how, decides your per-connection cost, your failure modes and where you look when a call reaches the wrong place.

For most of Kubernetes' history the component keeping that promise was kube-proxy, and the way it kept it was iptables. That is precisely what changed, and the change is visible in the two panels of the next figure.

Start with the conclusion, because it contradicts a great deal of published material: iptables is not how modern Kubernetes clusters route Service traffic. eBPF datapaths have displaced kube-proxy and iptables in most new production clusters, and Cilium is the canonical case.

The path being replaced is worth walking once, in order, because it explains its own obsolescence. A pod sends to a Service address, a virtual address with no interface behind it. The kernel hands the packet to iptables, which keeps its rules in ordered lists called chains. A first chain is read in order, holding one rule set per Service. A second chain picks a backend, with one more rule for each endpoint behind that Service. A third chain rewrites the destination address so the packet goes to the chosen pod. That is the work kube-proxy programmed on every node.

Nothing about that is wrong, and on a small cluster nothing about it is slow. The strain is structural. A rule list is read in order, so the per-packet work grows with the number of Services and endpoints in the cluster. Worse, a change is not an edit to one entry: the rule set has to be reprogrammed, so the time to converge after a deployment also grows with the size of the cluster. The same design that costs nothing at small scale becomes the dominant cost at large scale, with nothing having been misconfigured along the way.

The right-hand panel is the replacement. eBPF is a Linux kernel facility that runs small verified programs at defined points inside the kernel, so networking, security and observability logic can be added without changing kernel source or application code. Verified is a precise word here: before a program is allowed to run, the kernel checks that it terminates and stays within its bounds, which is what makes it safe to accept such programs from outside the kernel tree at all.

Applied to a Service call, the sequence collapses. The pod asks to connect to the Service address, using the same address and the same application code as before. An eBPF program runs inside the kernel at that moment, where the socket for the connection is being set up, before a packet has been built at all. One lookup in a backend map resolves the Service, and the key of that lookup is the Service itself rather than a position in a rule list. The packet then leaves already addressed to a backend pod.

Two properties follow, and they are the substance of the change. The resolution happens once at connection setup rather than as an ordered walk on the way past. And the lookup is a map, so its cost does not grow with the size of the cluster the way an ordered list does. The figure commonly cited for the combined effect is a 30 to 40 percent gain in datapath throughput.

Treat that figure the way this course treats every adoption and performance number. "Commonly cited" means it is a benchmark result, and a benchmark result carries a workload, a kernel version, a cluster size and a topology, none of which are yours. It is sound as a statement about the order of the effect and about why operators moved. It is not a number to quote at your own cluster without measuring your own cluster, and an architecture decision record that cites it as though it were a specification is making a claim it cannot support.

The operational consequence is sharper than the performance one. If your method for diagnosing a Kubernetes Service problem is to list iptables rules on a node, you may be reading an empty room and concluding that nothing is configured. The question that has to come first is which datapath this cluster runs, because the answer decides whether the evidence you are about to gather exists at all.

There is a second consequence of moving the datapath into a kernel program, and it is the one that reshapes segmentation. A program running where the socket is set up already knows which workload the connection belongs to, before any address has been chosen. Once the datapath knows the workload rather than only the number, policy can be written about workloads too.

Cloud segmentation is written as intent about workloads, and the datapath resolves that intent to whatever addresses those workloads happen to hold this minute. That single move is what separates it from everything module 23 covered, and it exists because the alternative stopped working.

The alternative is an address as the match key. A rule permitting one specific address to reach a database is correct for exactly as long as that workload keeps that address. When the workload is replaced, the rule is stale. That is merely useless. The dangerous case is the next one: the address is recycled to a different workload, and the rule now permits something nobody authorised, silently, with no change to the rule and therefore nothing for a review to notice. In an estate where workloads are replaced several times a day, address-based rules do not drift slowly. They rot.

Intent-based policy states the sentence you would actually say out loud: the checkout workload may reach the payments workload on one port, and nothing else may reach the payments workload at all. The mapping from that sentence to current addresses is the platform's job, refreshed as pods come and go. This is the same separation of intent from mechanism that the control-placement material used, and it is why the rule drift the previous module warned about bites hardest in exactly the environment that changes fastest.

A cluster running inside a VPC has two ladders of enforcement, and both are real. The provider's security groups sit on virtual interfaces and control what reaches the machines. The cluster's own network policy, enforced by the CNI, controls what reaches a pod from another pod, which is traffic the provider never sees because it may never leave the node. Neither ladder covers the other's ground. The common failure is not misconfiguration of either one, it is the quiet assumption that the other one is doing the work, so nothing is enforced between two workloads that share a node and share a security group.

Whichever ladder you write on, default-deny is the only stance that survives change. A policy that lists what is permitted and refuses the rest keeps meaning the same thing as workloads are replaced. A policy that lists what is forbidden has to be extended every time something new appears, and in an environment where new things appear hourly, it will not be. This is the principle from the zero-trust module carried inland: zero trust removed the assumption that being on the network is permission to use it, and cloud segmentation applies that same refusal between two workloads inside the same estate. Reachability is not authorisation, east to west as well as north to south, and lateral movement is precisely what an intent-based default-deny policy is there to prevent.

Once identity is the match key, the evidence changes shape too, and this is the fifth outcome. A conventional flow log records that one address talked to another address, over some protocol and port, for some duration. That was always a slightly awkward record, because addresses are how machines refer to each other and not how humans reason about systems. In a cluster where addresses live for minutes, it is close to useless on its own: by the time you read the record, the address means nothing.

Identity-aware flow data records the workloads instead. Hubble is the observability layer that ships with Cilium. It reads flow data from the eBPF programs already running in the kernel, so it can show which services talk to each other, which connections a policy dropped and where a failure sits, without adding anything to the applications. That last clause is the operational argument: there is no agent to deploy into each workload and no library for application teams to adopt, because the datapath was already handling every one of those connections.

A record of that kind reads as a sentence rather than as a pair of numbers: it names the workload that called, the workload it called, the port, and the verdict the policy returned. Somebody who knows the services but not the addressing can read it without translating anything first, which is the whole point of the change.

The practical difference shows up in the first question asked during an incident. With address-based records, the question is which rule matched, and answering it starts with reconstructing what each address was at the time. With identity-aware records, the question is which policy returned this verdict for this pair of workloads, and the record still means something a week later because a workload identity outlives the address it was using.

Be honest about the limit, because it is a real one. Flow records say who talked and what the verdict was. They do not say what was in the payload, and in an estate where the payload is encrypted, nothing at that layer will. That boundary is exactly what the next module is about: choosing which signal to reach for, given the question you are actually trying to answer.