The big picture: why energy data matters

Every time you switch on a light, boil a kettle, or charge your phone, a chain of events happens that you never see. Electricity is generated at a power station or wind farm, travels through high-voltage transmission lines, steps down through local distribution networks, and arrives at your home through your meter.

Alongside the electricity, there is an invisible river of data. Your smart meter records how much energy you use every 30 minutes. That data travels to the Data Communications Company (DCC), then onwards to your energy supplier so they can bill you, and to Elexon so the national electricity market can be settled correctly. A second river runs in parallel from substation SCADA telemetry to the network operator's planning systems and on into grid investment decisions. Both rivers depend on the same regulator and the same industry codes.

In everyday language, energy system data is any information created because energy is being generated, transported, or consumed. It includes everything from smart meter readings to the technical specifications of a transformer in a substation, from the price of electricity on the wholesale market to the geographic coordinates of an underground cable.

In the official regulatory world, the definition is precise. Ofgem, the energy regulator, changed it formally on 8 October 2024 so that the regulated scope is no longer limited to data needed for the operation of the system.

The definition matters because it determines which data falls under Ofgem's Data Best Practice Guidance, the rules that govern how energy data must be managed. Under the old definition, many valuable datasets were outside scope. The new definition brings them all in.

Great Britain's energy system generates an extraordinary volume of data. Five scale indicators bound the engineering challenge:

• 40 million smart meters each recording 48 readings per day produce 1.92 billion readings per day, roughly 700 billion per year. Under Market-wide Half-Hourly Settlement (MHHS, cutover July 2027), every one of these will flow through the entire settlement chain.
• Thousands of SCADA-monitored substations, each sampled every 2 to 10 seconds, produce approximately 69 million data points per hour for a large DNO. This data flows over air-gapped Operational Technology networks, physically separated from the internet.
• About 20 gas chromatographs at National Transmission System entry points continuously measure the exact calorific value of gas flowing into the national pipeline system, determining how every gas bill in GB is calculated.
• 13 authoritative categories covering 87 distinct data types, each with a defined producer, consumer, governing instrument, and sensitivity classification.
• Seven major industry codes (BSC, REC, SEC, Grid Code, DCUSA, CUSC, UNC), each governing specific aspects of data creation, flow, and use.

Managing this volume reliably, securely, and accurately is one of the greatest data engineering challenges in UK infrastructure.

The GB energy data landscape is defined by genuine, unresolved tensions. Understanding these helps a learner read every other module in this course in context. These are not academic debates. They affect how data flows, who can access it, and what happens to personal privacy.

Tension 1: Open data vs security risk

Ofgem's Data Best Practice Guidance presumes data should be open by default. SCADA telemetry from substations, however, is a national security concern. The 2015 Ukrainian power grid cyber attack showed what happens when operational data reaches the wrong hands. The Data Triage Playbook tries to square this circle, but the line between "open for innovation" and "restricted for security" remains contested.

Tension 2: Centralisation vs fragmentation

Nordic countries use a single data hub. Norway's Elhub handles all meter data in one hop. GB uses five separate platforms (DCC, Elexon DIP, ElectraLink DTS, RECCo, NESO). The Data Sharing Infrastructure (DSI) aims to connect them, but governance is unresolved. A single hub is simpler, but creates a single point of failure and raises governance questions about who controls it.

Tension 3: Consumer consent vs system need

Ofgem argues half-hourly data is needed for settlement (no individual consent required, it is a legal obligation under the BSC). The ICO argues it is personal data requiring explicit consent. As of early 2026, this debate remains unresolved. The Consumer Consent Service (CCS), being built by RECCo, will eventually formalise the rules.

Tension 4: Innovation speed vs regulatory pace

AI and digital twins can transform grid planning today. RIIO price controls operate on 5-year cycles, and code modification takes 12 to 18 months. The system data infrastructure risks being permanently one regulatory cycle behind the technology curve. RIIO-3 (starting April 2026) includes a substantial digitalisation baseline, but no DSI licence condition.

Tension 5: National standards vs local variation

CIM provides an international data model (IEC 61970). Each of the 14 DNOs implements it differently, with varying maturity levels. The LTDSprogramme is closing the gap. Stage 1.3 completed November 2025, Stage 2 targets May 2026, Stage 3 targets November 2026. Full interoperabilityremains years away.

This course has 15 modules across three stages:

1. Foundations (Modules 1 to 5): what energy data is, who produces it, how the physical network generates it, the complete 87-type taxonomy, and smart meters.
2. Applied (Modules 6 to 10): the data lifecycle from meter to market, the regulatory hierarchy, data governance, privacy rights, and settlement mechanics.
3. Practice and Strategy (Modules 11 to 15): international comparison, transformation programmes, digitalisation governance, CIM interoperability, and strategic planning.

Each module starts with a learning contract (the three outcomes you will achieve), includes inline knowledge checks, and ends with key takeaways. Each stage has a practice assessment you can use before attempting the timed stage test.