Which observability signals actually help

Module 17 placed security controls at the layer where threats form. This module applies the same discipline to observability: choosing signals based on the specific question you are trying to answer. The three foundational signal types are logs, metrics, and distributed traces, and they are not interchangeable.

A log is a timestamped record of a discrete event: a packet was dropped, a connection was refused, a user authenticated. Logs are high in detail and low in aggregation. They answer "what happened and when?" but become expensive to query at scale. A busy firewall can produce millions of log entries per hour; searching them during an incident under time pressure is difficult without pre-built indexes.

A metric is a numeric measurement sampled over time: packets per second, CPU utilisation, connection table fill percentage, error rate. Metrics are low in detail and high in aggregation. They answer "is this number changing?" and can be graphed and alerted on efficiently. The cost is loss of context: a spike in error rate tells you something is wrong but not which specific request failed or why.

A distributed trace follows a single request across multiple services, recording timing and outcome at each hop. Traces answer "where did this specific request spend its time?" They are essential for latency diagnosis in distributed systems but require instrumentation at every service that touches the request.

Network devices produce their own distinct signal types. The Simple Network Management Protocol (SNMP) is the oldest and most widely supported. An SNMP manager polls managed devices at regular intervals, reading counters from the Management Information Base (MIB): interface utilisation, error counts, neighbour state, hardware status. SNMP traps provide the reverse direction: a device sends an unsolicited notification when a threshold is crossed.

NetFlow is a flow export protocol developed by Cisco and standardised in RFC 3954. A network device observes traffic passing through it and produces flow records: source IP, destination IP, source port, destination port, protocol, byte count, and packet count for each distinct flow. NetFlow records are exported to a collector where they can be queried to answer questions like "which destinations is this server talking to?" or "what is consuming the most bandwidth on this link?" NetFlow does not capture packet content; it captures conversation metadata.

sFlow (RFC 3176) takes a different approach: it samples a fraction of packets (for example, one packet in every 1,000) and exports the sampled headers. This scales to higher link speeds where counting every flow would overwhelm the device, at the cost of statistical accuracy rather than exact counts. Both NetFlow and sFlow are more useful than SNMP for traffic pattern analysis because they show per-flow behaviour, not just interface totals.

OpenTelemetry (OTel) is a Cloud Native Computing Foundation (CNCF) project that defines a vendor-neutral API, SDK, and wire protocol for collecting logs, metrics, and traces. Before OTel, instrumenting an application meant integrating separate libraries for each signal type, each with its own format and export destination. Switching observability backends required re-instrumenting the application.

The OTel Collector is an agent that can receive telemetry in multiple formats (including Prometheus metrics, Jaeger traces, and OpenCensus data), process it, and export it to any supported backend. This means the application code calls OTel APIs; the routing decision (which backend receives the data) is a configuration decision, not a code change. For network engineers, the OTel semantic conventions define how network-related attributes (peer address, transport protocol, DNS query time) should be recorded in traces and metrics, making cross-service correlation possible.

Alert fatigue occurs when a system produces so many alerts that engineers stop treating them as meaningful signals. The typical cause is not a lack of metrics but a surplus of metrics without clear thresholds, combined with alerting on symptoms that have no direct remediation path. A CPU alert that fires every day because the system is working normally at high load is noise; it trains engineers to ignore alerts, which is exactly the condition that causes real incidents to go undetected.

The correct discipline is to start with the user-facing question: "Is the service responding within the agreed time?" Then trace backwards: which intermediate signal predicts or explains a user-visible failure? Alert on that signal, with a threshold that demands a human response when crossed. Infrastructure signals (disk, CPU) should be recorded as metrics for diagnosis but not alert unless they are on a direct causal path to a user-visible failure.

The Facebook outage also illustrated signal resilience: signals that run on the same infrastructure as the service they observe will fail alongside that service. External synthetic monitoring (scripted requests from independent vantage points) can detect user-facing failures even when all internal tooling is dark. Both are necessary.