Applied capstone: diagnose a slow website without guessing

The Applied stage has covered TCP, congestion control, QUIC, DNS, routing, NAT, IPv6 transition, TLS, and a systematic troubleshooting method. This capstone applies all of them to a single scenario requiring layer-by-layer reasoning, protocol-aware language, and a four-part diagnosis note that could be handed to another engineer.

The scenario is deliberately realistic. Multiple symptoms are present. Some are misleading. The diagnosis requires ruling out plausible but wrong explanations and identifying the specific protocol failure with evidence.

The walkthrough below keeps that shape: one stated symptom, six steps that each measure a single thing, and a note another engineer could act on without repeating the work.

Context. Users in the London office report that checkout.company.com loads slowly or times out, but only since 09:00 today. The site works fine from the New York office. The development team says they deployed no code changes overnight. The infrastructure team says no server restarts occurred. You are the first person on-call to investigate.

Step 1: State the symptom precisely. Slow or timeout on checkout.company.com from London since 09:00. New York is unaffected. No application or server changes. The problem is environment-specific and time-bounded.

Step 2: Check DNS from the affected location. You rundig checkout.company.com @10.20.1.5 (the London DNS resolver) from a London server. The response comes back in 3800 ms. The same query against Google's 8.8.8.8 takes 45 ms. The A record returned is the same in both cases (93.184.216.50). DNS is resolving correctly but slowly from the London resolver.

Step 3: Test TCP connectivity. You runcurl -v --connect-timeout 10 https://checkout.company.com from London. The DNS phase takes 3.8 seconds. After DNS resolves, TCP connects in 85 ms. TLS handshakes in 92 ms. The HTTP 200 response returns in 160 ms. Total: 4.2 seconds, of which 3.8 seconds is DNS.

Step 4: Isolate the DNS delay. The London resolver at 10.20.1.5 is adding 3.8 seconds to every lookup. You check whether this is specific to this domain. You rundig google.com @10.20.1.5 and get a 45 ms response (served from cache). You run dig checkout.company.com @10.20.1.5 a second time and get a 45 ms response. The TTL on the record is 300 seconds. The first lookup was slow; subsequent ones are fast. The resolver was not caching company.com records.

Step 5: Find the cache miss cause. You check the London resolver's configuration log. At 09:00, a resolver configuration update was deployed that flushed the DNS cache. Since 09:00, every first lookup for any company.com hostname has required a full recursive resolution from the authoritative server. The company's authoritative server is in the US. Each uncached lookup from London adds a full transatlantic round trip to resolution time, approximately 150 ms per delegation step. For a three-step resolution (root, .com TLD, authoritative), that is 450 ms per step plus processing.

Step 6: Write the diagnosis note. Symptom: checkout.company.com loads slowly (4+ seconds) from London since 09:00. Other offices unaffected. First failed function: the London DNS resolver (10.20.1.5) is returning slow (3.8 second) responses for uncached company.com records after a 09:00 cache flush. DNS for other cached domains is fast (45 ms). TCP and TLS are normal once DNS resolves. Evidence: dig checkout.company.com @10.20.1.5takes 3.8 seconds on first query, 45 ms on second query (TTL 300). The resolver cache was flushed at 09:00 per the configuration log. Next test: increase the TTL on company.com records from 300 seconds to 3600 seconds to reduce the frequency of cold-cache lookups, and confirm the resolver's configuration change was intentional.

Nothing in those six steps required a theory about what was wrong. Each step measured one thing and let the number choose the next test. The discipline shows most clearly in the explanations it never had to chase, which is what the next section sets out.

Several plausible wrong explanations were available in this scenario. The TLS handshake could have been the problem (it was not; 92 ms is normal). The application server could have been slow (it was not; HTTP response was 160 ms). A routing change could have added latency (no; traceroute showed a normal path). The London firewall could have been blocking traffic (no; TCP connected fine).

Each of these would have been tested before reaching the correct answer, had the diagnosis started at the wrong layer. By starting at DNS and measuring resolution time, the 3.8 second anomaly was visible in the first test. One data point pointed at the right function. The remaining investigation confirmed it.

This is what good troubleshooting produces: a short path from symptom to evidence to conclusion, with discarded wrong explanations listed explicitly. Rejecting a weak explanation is not wasted work. It is part of the diagnosis. The second scenario removes the comfort of a single dominant number: DNS, TCP, and TLS all pass their checks, and the response is still wrong.

A second scenario, for additional practice. Users report that an API endpoint returns garbled or truncated responses intermittently. The API is a JSON REST service over HTTPS.

What you observe. DNS resolution is normal. TCP connects successfully. TLS negotiates without error. The HTTP response arrives with a 200 status code. The response body is truncated: valid JSON begins, then stops mid-object. This happens on roughly 30% of requests.

The first failed function. This is not DNS (correct responses). It is not TCP connection (established). It is not TLS handshake (completed). The HTTP response arrives but the body is wrong. The failure is at the application layer, specifically in how the response body is being delivered.

Narrowing with evidence. You capture traffic with tcpdump. You observe that the HTTP response arrives in two TCP segments. The first segment carries the headers and part of the body. The second segment is never delivered. The client sees a TCP FIN from the server before the body is complete. The server is closing the connection before sending the full response.

Investigation leads to the cause. The web server has a misconfigured request timeout: 500 ms. The application sometimes takes 600 ms to generate the response. When it does, the server's timeout fires and the connection is closed before the response body completes. The client receives a partial body and a premature TCP FIN.

The diagnosis note. Symptom: JSON API returns truncated responses on approximately 30% of requests. First failed function: the server closes the TCP connection before the response body completes; TCP FIN observed mid-body in packet capture. Evidence: tcpdump shows TCP FIN from server after partial body, HTTP status 200 arrives but response is incomplete. Next test: check the server's request timeout configuration and measure the distribution of application response times to confirm some requests exceed the timeout.

Both scenarios ended at a component that a single command could interrogate: a named resolver, a named server timeout. Two branches of the same investigation do not end that tidily, and neither of them appears in a curl timing breakdown at all. They are worth walking before the stage closes.

Branch one: the delay is congestion, not a fault. Suppose the London measurements had come back with DNS at 40 ms, a clean TCP connect, and a first byte that arrives late only while the office backup window is running. No component is broken. Queues are building on a shared link, and every packet waits behind them. Loss, or an explicit congestion mark, is the signal that congestion control depends on to find the right sending rate, so a small amount of it in a capture is evidence of a working transport rather than proof of a fault. The test that separates the two is a latency measurement under load compared with the same measurement when the link is quiet: if idle latency is fine and loaded latency is not, more bandwidth will not fix it. That branch is developed in the congestion control and low latency module, which covers CUBIC, BBR, and the L4S approach to keeping queues short.

Branch two: the client is not using the resolver you tested. The first scenario rested on one assumption that no longer holds by default: thatdig against 10.20.1.5 asks the same question the browser asked. A browser or operating system configured for DNS over HTTPS sends its queries inside an HTTPS session to a resolver of its own choosing, so the corporate resolver never sees them, its logs stay empty, and its cache state is irrelevant to the user's experience. The diagnostic move is to establish which resolver the client actually used before drawing any conclusion from resolver-side evidence: check the browser's secure DNS setting, check the operating system's resolver configuration, and compare a resolution timed by the client with one timed from the shell. Encrypted DNS also changes what the network can see rather than making anyone anonymous, which is the subject of the DNS resolution in practice module.

Neither branch changes the method. Both change which measurement counts as evidence, which is the difference between a diagnosis that holds up and one that names the last thing anybody changed.

You have completed the Applied stage. You can now explain TCP, congestion control, DNS, UDP, QUIC, routing, NAT, and TLS with standards-aligned language. You have a systematic method for diagnosing request failures, a set of tools matched to each layer, and a four-part format for communicating diagnosis to others.

The Practice and Strategy stage builds on this foundation. It covers how to place security controls at the layer where the risk actually forms, choose observability signals that explain behaviour rather than just recording it, and use segmentation and packet capture deliberately rather than as first guesses.

If any module in this Applied stage felt uncertain, revisit it before moving on. The Applied stage practice (available from the course overview) covers every module in the stage and will show which areas need a second pass. The Foundations stage practice is also still available for the underlying vocabulary and protocol data units.