Getting the data to forecast electricity prices

Which prices to forecast?

Firstly, it is necessary to distinguish between different prices we might be trying to predict. The UK market has day-ahead hourly prices published at 11am, day-ahead half-hourly prices published at 3:45pm, intraday prices that you have to pay to view, and final imbalance prices that are published afterwards. Octopus’s retail prices, which I ultimately pay, are a function of the day-ahead half hourly prices.

The basic idea

My basic intuition (which needs testing) is that most of the variation in hourly day ahead prices can be explained by variation in ‘demand minus wind generation minus solar generation’. Other factors which play a part will be: how much dispatchable generation capacity (eg gas, nuclear, biomass) is available, what gas prices are, and how cheap power from France, Belgium and the Netherlands is.

Data Sources

The main source I have used is Elexon’s BMRS (, which has a lot of good openly available data, and an API. A second source I have used in National Grid ESO’s data portal (, which is currently in beta (the original source was ).

Defining demand

Before we get to estimating demand, we have to be clear on what we mean by it. For example, if a household uses 3kwh from their rooftop solar panels, was their demand 3kwh (the gross demand) or 0kwh (net demand)? Ideally we would measure gross demand and generation separately. Unfortunately, though, we don’t have any data about how much electricity was generated from rooftop PV each hour.

Estimating wind generation

Wind generation varies a lot. The biggest driver is how windy it is going to be in the locations where we have wind turbines (onshore and offshore). There will be occasions when we don’t generate as much as we could, because the transmission network can’t get that power where it is needed, however I this has much less impact on day-ahead prices than imbalance prices.

Estimating solar generation

Because most of the solar generation is included as a reduction in demand, it is very difficult to forecast it directly. I decided to make the assumption that for a given time of year and day, there was a fixed amount of electricity that would be generated on a clear day, and then I could reduce that on cloudy days.

Estimating demand

Gross demand is primarily driven by the day of the week, and the time of day. Demand in the UK is higher in winter, partly due to the need for heating on cold days. (In hot countries high temperatures can also lead to higher demand, however air conditioning does not currently play a significant role in the UK.) Finally, because we are measuring demand net of solar generation, we can expect net demand to be further reduced during sunny days, especially in summer.

Concluding Thoughts

I started this exercise with hope that the BMRS data would include the components necessary to form a reasonable 7 day forecast of hourly wind, solar and demand, sufficient to estimate prices. Unfortunately the data that isn’t as useful as I would have hoped, and is particularly lacking in historical data.



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Guy Lipman

Guy Lipman


Fascinated by what makes societies and markets work, especially in sustainable energy. Views not necessarily reflect those of my employer.