Logging and detection basics
By the end of this module you will be able to:
- Define what events must be logged for authentication, authorisation, and data access
- Design a structured log schema that supports automated correlation and search
- Explain how a SIEM collects, aggregates, and correlates events to produce alerts
- Write a basic Sigma detection rule for a named attack pattern
SolarWinds SUNBURST supply chain attack, March to December 2020
SUNBURST: nine months undetected due to logging gaps
In December 2020, FireEye (now Mandiant) disclosed that SolarWinds Orion, a network monitoring platform used by approximately 33,000 organisations, had been trojanised. The malicious update named SUNBURST had been distributed via SolarWinds' official software update channel from March to June 2020 and remained active in affected environments for an average of nine months before detection.
Among the affected organisations were multiple US federal agencies including Treasury, Commerce, and Homeland Security. Post-incident analysis found that SUNBURST deliberately suppressed its activity when it detected security tooling processes on the host.
Many victims' logging pipelines had coverage gaps: perimeter firewall logging was enabled but internal east-west (lateral movement) traffic between servers was not captured. The attacker moved between systems using legitimate administrative credentials obtained from Active Directory. Detection would have required correlation of unusual service account authentication times, new remote procedure calls between previously unconnected hosts, and volume anomalies in LDAP queries - precisely the kind of patterns a properly configured SIEM with thorough log coverage would surface.
What to log: the security event taxonomy
thorough security security logging covers four event categories per NIST SP 800-92 Section 3 and the OWASP Logging Cheat Sheet.
Authentication events: every attempt regardless of outcome. Successful login (user ID, timestamp, IP, authentication method, MFA outcome), failed login (user ID, timestamp, IP, failure reason as a code, not plain text), logout (session ID, duration), and credential changes.
Privilege and authorisation events: role assignment and revocation, access to administrative functions, authorisation failures (user X attempted to access resource Y: denied), and service account credential use.
Data access events: read access to sensitive data classes, create/update/delete on sensitive records, bulk export or download operations (volume threshold triggers), and file system access to sensitive paths.
System events: service start and stop, configuration changes, software installation and update, and certificate expiry and renewal.
Never log sensitive values in plain text: passwords, session tokens, credit card numbers, and API keys must never appear in logs. Log the fact that a credential was submitted, not the credential itself. In 2019, GitHub disclosed that some passwords had been stored in internal logs due to a misconfigured authentication debugging path, requiring a proactive password reset for all affected users.
With an understanding of what to log: the security event taxonomy in place, the discussion can now turn to structured logging, which builds directly on these foundations.
Structured logging
Structured logging emits every log entry as machine-parseable data (typically JSON) with named fields of defined types, rather than a free-text string. Structured logs enable fast search, aggregation, and automated correlation across millions of events. An unstructured log entry like:
2024-03-15 14:23:01 WARN Failed login for user admin from 203.0.113.45 attempt 4
becomes a structured JSON log:
{
"timestamp": "2024-03-15T14:23:01.000Z",
"level": "WARN",
"event_type": "authentication.failure",
"user_id": "admin",
"source_ip": "203.0.113.45",
"attempt_number": 4,
"service": "auth-api",
"trace_id": "abc-123-def"
}The structured version can be ingested by a SIEM, aggregated by user_id and source_ip, and automatically trigger an alert after five failures within sixty seconds. The OWASP Logging Vocabulary defines standardised event type names for common security events, enabling consistent field naming across teams.
With an understanding of structured logging in place, the discussion can now turn to siem architecture, which builds directly on these foundations.
SIEM architecture
A SIEM (Security Information and Event Management) platform collects log data from across an environment, normalises it into a common schema, applies correlation rules to identify attack patterns, and generates alerts for security operations teams. Modern SIEMs also provide long-term log storage for forensic investigation and compliance reporting.
The SIEM pipeline proceeds through four stages:
- Collect via agents or log shippers (Filebeat, Fluentd, CloudWatch Logs Agent).
- Aggregate and normalise by mapping heterogeneous formats into a common schema (Splunk, Microsoft Sentinel, and Elastic Security each have proprietary normalisation layers; OCSF is an emerging open standard).
- Correlate by matching detection rules across multiple event streams (five failed logins for the same account from different IPs within 60 seconds is a brute force indicator; a single failure is not).
- Alert by generating SOC incidents with enriched context.
The SUNBURST case illustrated the SIEM gap: perimeter logging was enabled but internal east-west traffic was not. SUNBURST moved between systems via legitimate credentials. Detection required correlation across authentication events, network connections, and LDAP query volumes, all from separate log sources. Without that correlation, each individual event was invisible.
With an understanding of siem architecture in place, the discussion can now turn to sigma detection rules, which builds directly on these foundations.
“Organizations should establish a log management infrastructure that ensures log data is available for analysis when needed, that logs are protected from modification and unauthorized disclosure, and that retention policies are in place and enforced.”
NIST SP 800-92, Section 3: Log Management Infrastructure
“Log management is the process for generating, transmitting, storing, accessing, and disposing of log data. Organizations should ensure they have a sound log management infrastructure that provides sufficient capacity and performance.”
NIST SP 800-92: Guide to Computer Security Log Management, Section 2 (2006)
Sigma detection rules
Sigma is an open, vendor-neutral format for writing detection rules that can be compiled to queries for Splunk, Elastic, Microsoft Sentinel, Chronicle, and other SIEM platforms. A single Sigma rule describes what to look for in log data; a compiler translates it to the target platform's query language.
Sigma rules use YAML syntax. A rule for SSH brute force specifies the logsource (product: linux, service: auth), a detection selection (event_type: authentication.failure, service: sshd), a condition (count of source_ip greater than 10 within 60 seconds), a timeframe, and false positive notes. The condition clause specifies the threshold. The logsource block tells the Sigma compiler which log type to query.
Rules should be tagged with MITRE ATT&CK technique identifiers (for example, attack.t1110.001 for Brute Force: Password Guessing). This enables a SIEM to map alerts to the ATT&CK matrix and track coverage across tactics.
Common misconception
“Retaining logs for the minimum required period satisfies both compliance and incident response needs.”
When an incident is discovered months after it occurred (SUNBURST was discovered nine months after initial infection), logs covering the initial compromise period may have been deleted. CIS Controls v8.1 Control 8.3 recommends a minimum of 13 months for security-relevant logs. Incident response planning should define a tiered strategy: hot storage (30 days) for active investigation, warm storage (12 months) for compliance, and cold archive (3+ years) for long-running investigations.
Common misconception
“Retaining logs for longer is always better for security.”
Unlimited log retention creates significant privacy and legal risk. Security logs often contain personal data about users' activities, authentication events, and access patterns. UK GDPR Article 5(1)(e) requires data to not be retained longer than necessary. Each log source should have a defined retention period justified by its operational use: authentication logs 90 days for threat investigation, firewall logs 30 days, application audit logs up to 7 years if needed for fraud investigation. Retain only what serves a documented security or compliance purpose, and document the retention period and justification.
A SIEM analyst sees that the same service account authenticated successfully to eight different internal servers between 02:00 and 03:00 within thirty minutes. The account normally accesses one server during business hours. No individual event triggered a rule. Which SIEM capability surfaced this pattern, and which MITRE ATT&CK tactic does it likely represent?
A SOC analyst receives 847 alerts over a shift. The SIEM has no alert prioritisation configured: every rule fires at the same severity. The analyst triages 60 alerts before the shift ends. The following morning, a ransomware infection is identified that started 18 hours before detection. The root-cause alert was alert number 723 in the queue. What is the primary process failure and how should alert management be redesigned?
A security team's authentication logs contain personal data. UK GDPR Article 5(1)(e) requires personal data to be kept no longer than necessary. PCI DSS v4.0 Requirement 10.7 requires 12 months of log retention. How should the team resolve this?
Key takeaways
- thorough security security logging covers authentication, authorisation, data access, and system events. Never log credential values, session tokens, or personal data in plain text.
- Structured JSON logs with named fields enable machine-readable ingestion, fast search, and automated SIEM correlation. Unstructured free-text logs cannot be reliably parsed at scale.
- A SIEM collects, normalises, correlates, and alerts. Correlation across multiple event streams is what makes lateral movement detectable: no single event in the SUNBURST case was individually suspicious.
- Sigma rules provide a vendor-neutral detection format that compiles to any major SIEM platform. Rules should be version-controlled alongside application code and tagged with MITRE ATT&CK identifiers.
- Log retention must satisfy both minimum legal requirements (PCI DSS: 12 months) and UK GDPR data minimisation. Pseudonymisation resolves most apparent conflicts between these requirements.
You can now design logging and detection for production systems. The Applied capstone brings together everything from threat modelling through detection: a single integrated scenario that tests whether you can design, verify, and monitor a secure system end to end. Module 16 is the Applied capstone.
Standards and sources cited in this module
NIST SP 800-92: Guide to Computer Security Log Management
Log collection, retention, and protection requirements including infrastructure and operational processes.
CIS Controls v8.1, Control 8: Audit Log Management
Prescriptive log management guidance including the 13-month retention recommendation.
Security event taxonomy, logging vocabulary, and structured field definitions.
Sigma Rule Repository (SigmaHQ)
Open-source library of Sigma detection rules with MITRE ATT&CK tagging.
MITRE ATT&CK Enterprise Matrix
Lateral Movement (TA0008) and Credential Access (TA0006) tactics referenced in detection rules.