A repeatable method for troubleshooting requests

Every request goes through a predictable sequence of functions. DNS resolves a name to an IP address. The operating system opens a TCP connection, or a QUIC connection for HTTP/3. TLS negotiates an encrypted channel (inside QUIC, that negotiation is part of the transport handshake rather than a separate step). The application protocol (HTTP, SMTP, etc.) sends the actual request. Each step depends on the previous one.

When a request fails, the method is: identify the first function in the chain that did not succeed. Test that function specifically. If it fails, that is where the fault is. If it succeeds, move to the next function. Do not change anything until you have confirmed which function failed.

This sounds obvious. In practice, most failures to diagnose correctly happen because someone skipped to a layer they assumed was the problem, or changed two things at once and lost the ability to know which change made the difference.

Testing one function at a time only works if you can interrogate each function on its own. Resolution, connection, handshake and request each have a command that answers one question about them and stays silent about the rest, which is what makes the answer usable as evidence.

Each layer has specific tools that produce specific evidence. Using the wrong tool at the wrong layer produces noise, not signal.

Layers 1 and 2 (Physical and Data Link). Check physical connectivity, interface status, and error counters. On Linux:ip link show shows interface state and error counts.ethtool eth0 shows physical link speed and duplex. On managed switches, check port statistics for CRC errors and input errors.

Layer 3 (Network). Use ping to test basic IP reachability. A successful ping proves the path to the IP address works at Layer 3. A failed ping may indicate routing, firewall blocking ICMP, or the host being down. Use traceroute (or tracerton Windows) to trace the hop path. Traceroute shows where in the path packets stop, but be aware that ICMP rate limiting and asymmetric routing can make the picture incomplete.

Layer 7 (Application), DNS. Use dig domain.comor nslookup domain.com to query DNS resolution. Include a specific resolver to test: dig domain.com @8.8.8.8 compares what Google's resolver returns versus your local resolver. Check for SERVFAIL, NXDOMAIN, or unexpected IP addresses.

Layer 4 (Transport). Use curl -v ortelnet host port to test TCP connectivity to a specific port. A "Connection refused" means the host is reachable but nothing is listening on that port. A timeout means the port is blocked (firewall) or the host is unreachable. A successful connection banner means TCP works; the problem (if any) is at a higher layer.

TLS layer. Use openssl s_client -connect host:443to test TLS. The output shows the certificate chain, handshake result, cipher suite, and any TLS errors. Look for "Verify return code: 0 (ok)" for success and error messages like "certificate verify failed" or "handshake failure" for failures.

Application layer (HTTP). Use curl -v https://host/pathfor a complete end-to-end test. The verbose output shows DNS resolution, TCP connection time, TLS handshake details, HTTP headers, and response code. A 200 means success. 4xx means a client-side request problem. 5xx means the server errored. A redirect loop shows as repeated 3xx responses.

Packet capture. Use tcpdump or Wireshark for full packet visibility when other tools are insufficient. Capture is intrusive and produces large amounts of data. Use it when you need to see exactly what is on the wire, for example to confirm retransmissions, zero-window events, or unexpected RST packets.

Worked example: one site, two networks, two protocols.A colleague reports that a site is slow from the office but fine from their phone's hotspot. Nothing has failed, so there is no error message to read. The complaint is a comparison, which means the first job is to make the comparison exact: run the same request from both networks and record what each one actually negotiated.

The browser developer tools network panel has a Protocol column, and an HTTP/3-capable build of curl will answer directly: curl --http3-only -sI https://example.com returns headers where HTTP/3 is reachable and an error where it is not. On the hotspot the answer is h3. On the office network the same request to the same site returns h2. The site advertises HTTP/3 identically to both: the response carries an Alt-Svc header naming h3, and the DNS HTTPS record says the same thing.

HTTP/3 carries HTTP over QUIC, and QUIC runs over UDP, in practice on port 443 (RFC 9114). A firewall that permits TCP 443 and drops UDP 443 blocks nothing the user can see. The client's QUIC attempt gets no reply, the attempt times out, and the browser falls back to HTTP/2 over TCP on the port that is open. That is a protocol downgrade, not an outage, and nothing logs it.

Two cheap tests confirm it. Capture on the client with the filter udp port 443: outbound QUIC Initial packets with nothing coming back is the signature of a silent drop, whereas an ICMP administratively-prohibited reply names the device doing it. Then read the egress rule set for a UDP 443 permit. If the rule is there and the capture is still one-sided, the drop is upstream of you, and that is what the diagnosis note escalates.

The downgrade is the finding, not yet the fault. HTTP/2 over a healthy path is not slow, and blocking UDP 443 is sometimes a deliberate policy so that traffic stays inspectable. What the evidence removes is the false comparison: while one network runs h3 and the other runs h2, the office and the hotspot are not measuring the same thing, and any judgement about the site is partly a measurement of the firewall.

That example turned on reading a one-word answer, h2 or h3, exactly as it was given. Network tools are precise about failure in the same way, and the distinctions they draw are most of the diagnosis.

Error messages from network tools are precise. "Connection refused" is not the same as "Connection timed out." "SERVFAIL" is not the same as "NXDOMAIN." Reading them accurately is part of the diagnosis.

Connection refused (TCP RST). The destination host is reachable. No service is listening on the target port, or a firewall on the host itself is actively refusing the connection. The host responded with a TCP RST.

Connection timed out. Packets are being dropped without a response. Either a firewall is silently dropping packets, the host is unreachable, or routing is blackholing traffic. No RST was received.

DNS SERVFAIL. The recursive resolver attempted to answer but encountered a failure, often because the authoritative server was unreachable or returned an error.

DNS NXDOMAIN. The authoritative server confirmed the domain does not exist. This is a definitive answer, not a failure to reach the server. Treat NXDOMAIN as evidence about the name itself, not about the path to the server that answered.

TLS certificate verify failed. The certificate chain could not be validated. Common causes: missing intermediate certificate, expired certificate, hostname mismatch, or untrusted CA.

None of these distinctions survive being retold as "the site is down". A refused connection, a timed-out one and a negotiated h2 where h3 was expected send the work to three different teams, so the wording has to travel intact to whoever acts on it next.

A diagnosis note has four parts. Symptom: what was observed, by whom, and when. First failed function: which layer failed, with the specific protocol and command output. One observable: the exact command and its relevant output. Next safe test: one specific action that would confirm or narrow the diagnosis.

Example. A developer reports an API endpoint is returning no response.

Symptom: POST https://api.company.com/orders returns no response after 30 seconds from the production application server (10.2.1.5).

First failed function: TCP connection to api.company.com:443 times out from 10.2.1.5. DNS resolves correctly (93.184.216.34). The failure is at Layer 4.

Observable: curl -v --connect-timeout 5 https://api.company.comfrom 10.2.1.5 returns "Connection timed out" after 5 seconds. The same command from the developer's laptop succeeds in 0.3 seconds.

Next safe test: Check firewall rules between VLAN 200 (where 10.2.1.5 lives) and the internet for outbound port 443. A recent change log entry may identify a new rule that dropped this path.

Anyone reading this note knows the symptom, where it fails, the evidence, and what to look at next. No expertise is required. No time is wasted checking the application code.

A note in that shape is already an escalation. The firewall team receives a confirmed layer, a command they can rerun and a rule to inspect, so the handover costs them nothing to re-derive. What remains is knowing when the note should leave your hands at all.

Escalate when you have confirmed which layer failed but do not have access to fix it. You have verified the fault is in the firewall configuration, but only the network team can change firewall rules. Escalate with the diagnosis note, not with the symptom. "Firewall between VLAN 200 and internet is blocking port 443, confirmed by traceroute and curl timeout" is an escalation. "The internet is down" is not.

Escalate when you have reached the limit of your visibility. You can see that traceroute stops at a specific hop but cannot tell whether that hop is a misconfigured router or a carrier network issue. The diagnosis note records what you can see and where the evidence ends.