Practice capstone: when dashboards look healthy and users disagree

The practice stage has placed controls by layer, moved remote access from network reachability to per-request authorisation, contained blast radius with segmentation, rebuilt all of it in cloud and overlay networks, chosen which signals to trust, used captures deliberately, seen protocol features turned into attack surfaces, and pushed content out to the edge. This capstone pulls those together with the layered model from Foundations and the protocol behaviour from Applied. The scenario is designed to require reasoning from all of them.

The scenario: it is 14:30 on a Tuesday. Your on-call phone rings. A product team reports that the checkout flow on your e-commerce platform is returning errors for a subset of users; roughly 30 percent of checkout attempts are failing with a 503 status. Your infrastructure dashboards show CPU utilisation at 18 percent across all application servers, network bandwidth at 12 percent of capacity, and no firewall drop alerts. The application server health checks are passing. The load balancer shows all backend instances as healthy.

This is not a resource exhaustion problem at the infrastructure layer. Something else is wrong. What you observe is a symptom. The question is: at which layer, and in which component, is the actual failure?

The symptom is a 503 response on checkout. HTTP 503 means "Service Unavailable": the server understood the request but cannot fulfil it right now, typically because a dependency is unavailable or a resource limit is reached. This is a Layer 7 signal. Start there, then work toward the cause.

Layer 7 question: What is the application server's error log saying? Application logs are the right signal here: they record the event with context. If the logs show "connection pool exhausted" or "upstream timeout from payment-service," the layer of failure is now identified as an application dependency. The application server itself is healthy; it cannot reach something it needs.

Assume the logs show: "upstream connect error or disconnect/reset before headers, upstream: payment-service:8080." The application is returning 503 because it cannot reach the payment service. This is no longer a question about the application server; it is a question about connectivity from the application tier to the payment service.

Layer 4 question: Can the application server establish a TCP connection to the payment service on port 8080? A simple telnet or nc (netcat) test from the application server to payment-service:8080 answers this. If TCP does not complete the handshake (connection refused, or timeout), the failure is at Layer 4 or below. If TCP connects but the payment service returns an error response, the failure is at Layer 7 within the payment service.

Assume TCP connection is refused. The application server cannot reach port 8080 on the payment service. The symptom has moved from the application tier to the connectivity between tiers.

A refusal is a narrow piece of evidence but not a unique one. Exactly three conditions produce it on this path, and separating them is the difference between restarting the wrong service and restoring one rule.

Connection refused means one of three things: the payment service process is not listening on port 8080; a firewall or security group is blocking the connection; or the IP address being dialled is incorrect. Check each in turn.

Is the payment service listening? On the payment service host, check which ports are listening: ss -tlnp or netstat -tlnp. If port 8080 is not in the output, the process is down or misconfigured. If it is listening, the issue is network-side.

Assume port 8080 is listening on the payment service. The process is running. The failure is in the network path between the application tier and the payment service. Check the segmentation: are these two services in the same network zone or in different zones separated by a firewall or a security group? If they are in different zones, check the inter-zone firewall rules or security group rules for a rule permitting the application tier to reach the payment service on port 8080.

A security group audit reveals that a deployment one hour earlier changed the payment service's security group inbound rules. The rule permitting inbound TCP 8080 from the application tier security group was accidentally deleted. The payment service is running and healthy; the connection is being dropped by the security group before it reaches the service. Adding the rule back restores service within seconds.

That is one path through the evidence, and it closed quickly because the first branch tested happened to be the right one. The same opening evidence, healthy infrastructure and a failing subset of users, supports two other branches this walk never had to take. Both are common in the architectures the practice stage has just built, and neither is visible to a connectivity test.

Start both branches with one question: what distinguishes the failing 30 percent from the working 70 percent? If the answer names something on the server side (one instance, one availability zone, one dependency), the walk above applies. If the answer names a population of users, or a geography, the walk above will keep returning healthy results while users keep failing.

Branch one: is the user authorised at the proxy? If the failing users share an attribute (they are contractors, they are on unmanaged laptops, they were moved into a new group last week), the question is no longer whether packets arrive. Under zero trust, being on the network is not permission to use an application. The request passes through an identity-aware proxy that evaluates identity, device state and context on every request, so a policy edit or an expired device-compliance signal denies one set of users while every path, route and rule stays healthy.

The observable is the proxy decision log: it records who was denied, which policy decided it, and on what input. Read that before the HTTP status, because the status is whatever the front end chose to return, which may be a redirect to a sign-in page, a 403, or an error page indistinguishable from a backend failure. Note also that a test from a shell on the application server cannot reproduce this branch at all: that request carries no user identity, so it is not the request that failed.

Branch two: is the failure at the edge or at the origin? If the platform is fronted by a CDN, then the server is at least two servers: an edge node in a point of presence near the user, and the origin behind it. Both can return a 503, and the user cannot tell them apart.

Fetch the same path twice: once through the public hostname, once directly against the origin with the CDN bypassed. Compare the status and the response headers that name the serving node and the cache state. Origin healthy while the edge fails puts the fault at the edge or on the edge-to-origin path; both failing makes the edge a messenger and sends you back to the walk in section 29.3. Because anycast routes each user to a nearby point of presence, an edge fault is usually regional, so group the failing requests by point of presence or region before deciding that the affected users have anything else in common.

In this incident neither branch was the answer, and that is exactly why the note has to record them. A reader who cannot tell an eliminated hypothesis from an unconsidered one will start the next incident by testing what you already ruled out.

A diagnosis note is not an event log. It is a structured record that another engineer can read and validate. It covers: what was observed (the symptom and its scope); what was tested (the observables used at each step); what was found (the layer and component of failure); what was changed (the remediation action); and what confirmed resolution (the observable that showed the fix worked).

A well-written diagnosis note for this scenario would read: "At 14:30, approximately 30 percent of checkout requests returned 503. Application server logs showed upstream connection errors to payment-service:8080 from 14:12 onward. TCP connection test from app-server-1 to payment-service:8080 returned connection refused despite payment service listening on 8080 (confirmed via ss on the payment host). Access proxy decision log showed no denials for the affected users between 14:00 and 14:45, and requests made directly to the origin returned the same 503 as requests through the CDN edge, so authorisation and edge faults were eliminated. Security group audit identified that a deployment at 14:08 removed the inbound rule permitting TCP 8080 from app-tier-sg. Rule was restored at 14:52. Error rate returned to baseline within 90 seconds. No data loss confirmed; checkout transactions failing with 503 were not partially committed (payment service logs confirm no partial transactions)."

Notice what the note does not include: "the network was slow," "something went wrong with the deployment," or "it seems like maybe the firewall." Every statement in the note is an observable or a directly confirmed fact. This is what "technically disciplined under pressure" means.