Hands on with market data: Insights, IRIS and the open APIs

GB market data is genuinely open, and that is unusual. Four platforms carry most of what a working analyst needs, and the useful way to tell them apart is not by who owns them but by two properties: cadence, meaning how fast the data arrives and how often it changes, and contract, meaning what the licence lets you do with it and how stable the interface is. Two platforms with the same owner can sit in different quadrants, and two platforms with different owners can behave almost identically. Pick by cadence and contract, and you will pick correctly.

The first is the Elexon Insights Solution. It publishes the settlement and balancing data that the Balancing and Settlement Code obliges Elexon to report: system prices per Settlement Period, generation by fuel type, demand outturn, balancing acceptances, and the market messages that sit around them. It exposes them as a REST API over HTTPS, returning JSON, with no account and no key required for the public datasets. That last point matters more than it sounds: a data source you can call from a notebook in ten seconds gets used, and a data source behind a registration wall does not.

The second is IRIS, Elexon's push service. Same underlying data, opposite delivery model. Instead of you asking for a dataset on a timer, IRIS delivers files to a subscriber as soon as they are published, and Elexon maintains open-source client libraries so that consuming the stream is not a bespoke engineering project. IRIS exists because polling a REST endpoint every few seconds is a poor way to catch data that arrives irregularly.

The third is the NESO Data Portal. This is a catalogue rather than a single API. NESO publishes hundreds of datasets there, from historic generation mix to embedded capacity information, constraint costs and forecasting inputs. Each dataset carries its own licence, and most of them are published under a CC BY 4.0 licence, which permits reuse including commercial reuse as long as you attribute the source. The word “most” is doing real work in that sentence, and section 16.2 makes you check.

The fourth is the Carbon Intensity API at carbonintensity.org.uk, operated by NESO with the University of Oxford, EDF Europe and WWF. It serves national and regional carbon intensity in grams of CO2 per kWh, both outturn and forecast, and it is the most consumer-facing of the four: it is what sits behind carbon-aware appliance scheduling, low-carbon charging apps and a good deal of research. The API is at api.carbonintensity.org.uk and the documentation at docs.carbonintensity.org.uk.

These four are not the whole estate. Suppliers publish their own open interfaces, and the Octopus developer API is the best known: it exposes product and tariff information, including half-hourly unit rates for the Agile tariff, without authentication for the public endpoints. Distribution network operators publish their own open data portals as well, which the-wires-and-pipes and flexibility-market-data both draw on. The pattern that runs through all of them is the presumed open principle from Ofgem's Data Best Practice: a non-personal energy dataset should be open unless the custodian has evidence for restricting it.

Hold the four in a quadrant. Down one axis, cadence: near real time against periodic. Across the other, contract: a fixed reporting obligation with a versioned interface against a catalogue where each dataset sets its own terms. The figure places them.

The takeaway is a habit rather than a fact. Before you write a line of code, decide which quadrant your question lives in. A question about what the market did in a specific Settlement Period is an Insights question. A question that needs to react within seconds of publication is an IRIS question. A question about a bounded historic or planning dataset is a Data Portal question, and the first thing you read there is the licence. A question about how clean the electricity will be in Cornwall at four o'clock tomorrow is a Carbon Intensity question.

This section and the next are guided walkthroughs. They describe the request you make, the shape of what comes back and how to read it, so that you can reproduce each one yourself with a browser, curl or a few lines of Python. Nothing on this page calls an external API for you, deliberately: a course page that depended on a live third-party service would break in ways that have nothing to do with your learning.

Walkthrough 1: yesterday's system price and generation mix

Start with the Insights API documentation at bmrs.elexon.co.uk. It lists the datasets by name with their parameters, and the developer portal at developer.data.elexon.co.uk gives you the machine-readable specification. Work from those pages rather than from a URL you found in a forum, because dataset paths are versioned and forum URLs age badly. Pick the balancing system price dataset and the generation by fuel type dataset, and set the date range to a single completed settlement day.

What comes back is JSON with a small envelope and an array of records. Each system price record is keyed by settlement date and Settlement Period, one to 48 or 46 or 50 depending on the clock change, and carries the imbalance price for the period along with the volume and the run that produced it. Two things trip people up here. First, the period is not a clock time: period 1 begins at 23:00 UTC on the previous day for a summer date, and if you plot periods as if they were hours you will produce a chart that is quietly wrong. Second, the same settlement date can be requested again later and give you a different number, because the settlement run that produced it has moved down the ladder you learned in from-meter-to-market. Record which run your figure came from.

The generation by fuel type response is a similar array, one record per fuel type per interval, with a megawatt value. Sum the fuels for an interval and you have the metered transmission-connected output, which is not the same as GB demand: embedded solar and embedded wind sit below the transmission boundary and appear as reduced demand rather than as generation. That single distinction explains most of the confused comparisons you will see between different published mix figures.

With those two arrays joined on settlement date and period, you can plot price against mix for the day, and the shape will teach you something the theory cannot: the periods with the highest prices are usually the ones where the wind share collapsed, and the price does not simply track demand.

Walkthrough 2: a NESO Data Portal treasure hunt with a licence check

Now go to the NESO Data Portal and find a dataset you would actually use, for example historic generation mix or a constraint cost series. The portal gives each dataset a landing page with a description, a preview, a download in CSV and often a query interface. The discipline this walkthrough teaches is what you do before the download button.

Read the licence statement on the dataset page. Most NESO datasets are published under CC BY 4.0, which allows you to copy, redistribute, adapt and build on the data for any purpose including commercial, provided you give appropriate credit and indicate whether you made changes. Some datasets carry different terms, and a handful are restricted. Then read the update frequency and the coverage dates. A dataset that stopped being refreshed two years ago is not a live feed no matter how current the download date looks, and this is the single most common way an analysis built on portal data goes wrong.

Write down four things before you use any portal dataset: its exact title, its publisher, its licence, and the date you accessed it. That is the attribution CC BY asks for, and it is also what makes your own work auditable. The open data movement runs on attribution rather than on payment, and treating attribution as an afterthought is both a licence breach and bad practice.

The two walkthroughs in this section answer a question a consumer might genuinely ask: when should I run the washing machine? One answers it in carbon, the other in money, and the fact that the two answers often differ is the lesson.

Walkthrough 3: the regional carbon intensity forecast

The Carbon Intensity API publishes both national and regional figures. The regional service divides GB into fourteen regions aligned to the distribution network boundaries. These are not the same as the fourteen GSP Groups you met in identifiers and registers, even though the counts match, so do not join the two on region name. It publishes a forward forecast running more than 96 hours ahead in half-hourly resolution. Request a region by its identifier and a forward window, and you get an array of half-hourly records.

Each record carries a start and end timestamp, a forecast intensity in grams of CO2 per kWh, an index band that turns the number into a plain-language label, and a breakdown of the expected generation mix by fuel as percentages. Reading it is straightforward once you notice two things. The regional numbers differ sharply from the national one, because a region with a large wind fleet behind it looks nothing like a region fed mostly by gas. And forecast is not outturn: the further ahead you look, the wider the error, so a forecast three days out tells you the shape of the week rather than the exact half-hour.

To find the greenest half-hour, sort the array by forecast intensity and take the minimum, then check whether it falls at an hour a household would tolerate. Do that across a week and the pattern is usually blunt: overnight and early afternoon are clean, the early evening peak is dirty, and the size of the gap depends far more on wind than on anything a consumer controls.

Walkthrough 4: Agile unit rates and the cost of a wash cycle

Now price the same question. The Octopus developer API publishes product and tariff data openly. Find the Agile product, then the tariff code for your region, then request the standard unit rates for a date range. Agile is a time-of-use tariff whose unit rate changes every half-hour and is published for the following day in the afternoon, so a request made in the evening returns rates covering the rest of today and all of tomorrow.

Each record in the response has a start time, an end time and a unit rate, given both with and without VAT. Take a wash cycle drawing about 1 kWh over a bit under an hour and cost it at three different start times: the overnight trough, mid-afternoon, and 17:30 in the evening peak. The spread between the cheapest and the most expensive half-hour on a windy winter day is large enough to be worth acting on, and on rare days the overnight rate goes negative, which means the tariff pays you to consume.

Then lay the two walkthroughs side by side. The cheapest half-hour and the cleanest half-hour are correlated, because both follow wind, but they are not the same half-hour: price also carries network charges, policy costs and the balancing position of the day. Anyone building a consumer-facing optimiser has to decide which of the two they are optimising, and say so, because a device that quietly optimises cost while advertising carbon is misleading its user. That question comes back in devices-and-tariff-data, where the standards for controllable appliances live.

Everything so far has been pull: you decided when to ask, and the platform answered. Pull is the right default, and it is also the wrong model for a class of problems. If data is published irregularly and you need it within seconds, polling forces an unpleasant trade-off between wasted requests and missed updates. Push inverts the relationship: you register once, and the publisher sends you each file as it is produced.

IRIS is Elexon's push channel for exactly this. A subscriber is provisioned with credentials, chooses the datasets it wants, and receives published files as they appear. Elexon maintains open-source client libraries on GitHub, so the integration work is configuration rather than protocol implementation. The design consequence worth internalising is that a push consumer needs somewhere to put what arrives: push gives you delivery, not storage, and a subscriber with no landing store will lose data the first time its process restarts.

The two models are not competitors. A realistic near-real-time dashboard uses both: IRIS for the live tick, and an Insights REST call for backfill after an outage and for any historic window the stream has already passed. Build the store first, feed it from the stream, and let the REST API repair the gaps. The figure sets the two side by side.

The takeaway: choose pull when your question has a defined time window, and push when your reaction time is measured in seconds. Anything that has to react to a publication rather than to a clock wants push.

Estimates as data: PV_Live

Not every dataset is a measurement. Most GB small solar is connected below the transmission boundary and is therefore invisible to the system operator's metering. Their output still has to be known, so it is estimated. PV_Live, run by Sheffield Solar with NESO, estimates national and regional solar generation half-hourly from a sample of monitored sites scaled to the installed capacity base, and NESO uses it in operational forecasting.

Treat estimated data as a distinct class. It has an uncertainty band, it is revised after publication as more sample data arrives, and it depends on a capacity register that is itself imperfect. None of that makes it unusable; it makes it something you have to label. If you publish a chart mixing metered Insights output with estimated embedded solar, say which series is which. The same distinction underlies the export register teaching in export-and-microgeneration: exported energy sits on a separate register against an export MPAN, and where no such register exists, the volume is estimated rather than measured.

Transparency as an obligation: REMIT

One last category of open data is open because the law requires it. REMIT, the regulation on wholesale energy market integrity and transparency, obliges market participants to publish inside information, such as an unplanned outage at a large generating unit, before trading on it. The result is a public stream of outage and availability notices that sits beside the reporting platforms in this module. It is a useful reminder that openness in this sector has three different sources: a licence obligation on a reporting agent, a policy principle such as presumed open, and a market-integrity rule that makes disclosure a condition of trading.