Logging and detection basics

Thorough security logging covers four event categories per NIST SP 800-92 Section 3 and the OWASP Logging Cheat Sheet.

Authentication events: every attempt regardless of outcome. Successful login (user ID, timestamp, IP, authentication method, MFA outcome), failed login (user ID, timestamp, IP, failure reason as a code, not plain text), logout (session ID, duration), and credential changes.

Privilege and authorisation events: role assignment and revocation, access to administrative functions, authorisation failures (user X attempted to access resource Y: denied), and service account credential use.

Data access events: read access to sensitive data classes, create/update/delete on sensitive records, bulk export or download operations (volume threshold triggers), and file system access to sensitive paths.

System events: service start and stop, configuration changes, software installation and update, and certificate expiry and renewal.

Never log sensitive values in plain text: passwords, session tokens, credit card numbers, and API keys must never appear in logs. Log the fact that a credential was submitted, not the credential itself. In 2019, GitHub disclosed that some passwords had been stored in internal logs due to a misconfigured authentication debugging path, requiring a proactive password reset for all affected users.

Structured logging emits every log entry as machine-parseable data (typically JSON) with named fields of defined types, rather than a free-text string. Structured logs enable fast search, aggregation, and automated correlation across millions of events. An unstructured log entry like:

2024-03-15 14:23:01 WARN Failed login for user admin from 203.0.113.45 attempt 4

becomes a structured JSON log:

{
  "timestamp": "2024-03-15T14:23:01.000Z",
  "level": "WARN",
  "event_type": "authentication.failure",
  "user_id": "admin",
  "source_ip": "203.0.113.45",
  "attempt_number": 4,
  "service": "auth-api",
  "trace_id": "abc-123-def"
}

The structured version can be ingested by a SIEM, aggregated by user_id and source_ip, and automatically trigger an alert after five failures within sixty seconds. The OWASP Logging Vocabulary defines standardised event type names for common security events, enabling consistent field naming across teams.

A SIEM (Security Information and Event Management) platform collects log data from across an environment, normalises it into a common schema, applies correlation rules to identify attack patterns, and generates alerts for security operations teams. Modern SIEMs also provide long-term log storage for forensic investigation and compliance reporting.

The SIEM pipeline proceeds through four stages:

1. Collect via agents or log shippers (Filebeat, Fluentd, CloudWatch Logs Agent).
2. Aggregate and normalise by mapping heterogeneous formats into a common schema (Splunk, Microsoft Sentinel, and Elastic Security each have proprietary normalisation layers; OCSF is an emerging open standard).
3. Correlate by matching detection rules across multiple event streams (five failed logins for the same account from different IPs within 60 seconds is a brute force indicator; a single failure is not).
4. Alert by generating SOC incidents with enriched context.

The SUNBURST case illustrated the SIEM gap: perimeter logging was enabled but internal east-west traffic was not. SUNBURST moved between systems via legitimate credentials. Detection required correlation across authentication events, network connections, and LDAP query volumes, all from separate log sources. Without that correlation, each individual event was invisible.

A SIEM does not usually fail by missing a rule. It fails when it produces more alerts than anyone can read. Alert fatigue is the operational condition where analysts face so many notifications, most of them false positives or low value, that genuine signals are acknowledged late, dismissed without investigation, or missed entirely. It is the central operational failure of detection at scale, and it is usually caused by the instinct that feels most responsible: adding more rules.

More rules is not more detection. Every rule that fires without leading to action trains the analyst to treat the queue as noise, so the one alert that mattered arrives in the same undifferentiated stream as the thousand that did not. A SIEM tuned for coverage, where every conceivable pattern has a rule at the same severity, detects less in practice than a smaller set of high-fidelity rules an analyst trusts. The work of detection is therefore as much about retiring noisy rules, tuning thresholds, and assigning severity tiers as it is about writing new ones. Detection quality is measured by the ratio of actioned alerts to total alerts, not by the count of rules deployed.

The SIEM is also no longer a standalone category. It now sits inside a convergence with SOAR, which automates the response steps that follow an alert, and XDR, which correlates detection across endpoint, identity, network, and cloud telemetry rather than log streams alone. For this module the SIEM is the collection, normalisation, and correlation engine; how SOAR and XDR extend it into an operational capability is defined in the later practice module on detection, response and security operations.

Sigmais an open, vendor-neutral format for writing detection rules that can be compiled to queries for Splunk, Elastic, Microsoft Sentinel, Chronicle, and other SIEM platforms. A single Sigma rule describes what to look for in log data; a compiler translates it to the target platform’s query language.

Sigma rules use YAML syntax. A rule for SSH brute force specifies the logsource (product: linux, service: auth), a detection selection (event_type: authentication.failure, service: sshd), a condition (count of source_ip greater than 10 within 60 seconds), a timeframe, and false positive notes. The condition clause specifies the threshold. The logsource block tells the Sigma compiler which log type to query.

Rules should be tagged with MITRE ATT&CK technique identifiers (for example, attack.t1110.001 for Brute Force: Password Guessing). This enables a SIEM to map alerts to the ATT&CK matrix and track coverage across tactics.

A single rule is where detection starts, not where it ends. The later practice module on detection, response and security operations treats these rules as detection-as-code: version-controlled, tested, peer-reviewed, and shipped through a pipeline in the same way as application code, so the rule set is maintained as a living capability rather than a one-off file.