Forecasting UK electricity prices

I wrote a blog post a couple of weeks ago discussing the challenges of getting the data to forecast hourly day-ahead electricity prices for Great Britain. I concluded that it was always going to be a challenge, especially to verify how well any approach was likely to work. That wasn’t meant to suggest that the challenge was impossible, and this blog post describes a method that produces forecasts, although I can’t yet report how good they are.

I have implemented this model in a python on a server. You can see the code in github and the results at https://energy.guylipman.com/forecasts.

Forecasting demand

As noted in the previous blog post, we have to distinguish between gross demand (how much customers use), and reported demand (how much is used, less small-scale generation which is mostly solar). I have historical reported demand and historical solar production, and I find that in summer the gross demand at a particular time of the day / day of the week is quite similar from week to week. In winter temperature will provide an added complication, but I will think more about that when it arrives.

As a result, I have assumed that the demand for an upcoming weekday/time will be the same as it was on that weekday/time in the most recent week. For each day in the past 7, therefore, I extract the reported demand from BMRS report FORDAYDEM, type DANF. I extract the solar generation from these days from BMRS report B1630, and add it back to the demand to get the gross demand for each weekday/time. I haven’t implemented handling of public holidays, but presumably these will follow some combination of a Sunday and a weekday profile.

This method will inherently have a bit of a lag — for example, as we approach winter, we would expect gross demand to gradually rise, however I feel that being a week behind won’t be too much of a problem and allows us to pick up fundamental shifts in demand like we have seen from Covid lockdowns. A bigger concern is that this approach won’t take into account demand changes that would have been predictable to a fundamental analyst. However, applying this kind of insight requires ongoing judgement, so is beyond the scope of my project.

Forecasting solar generation

I have historic solar generation from BMRS report B1630, and I extract the past 14 days worth. The shape of each day’s generation is quite stable, although some days, presumably cloudy ones, have considerably lower generation. What I have decided to do is to treat the day with maximum generation over the past 14 days as the ‘true solar’. I then reduce the true solar according to weather forecasts of how sunny it is likely over each of the next 14 days. At the moment I am using 5 day UV forecasts for London from the UK Met Office, but I’m not entirely happy with this adjustment, so I am looking to revisit it. Having said this, forecast cloudiness has less impact than variation in wind and demand.

I chose to use 14 days of history to increase my chance of having a clear day for my true solar, but also to minimise the lag as the days get longer or shorter. Ideally I would also have access to UV history, which would be useful in understanding how much to adjust future days. I also made a conscious decision not to use solar generation data from previous years, given the rapid roll-out of solar capacity.

Forecasting wind generation

BMRS have hourly forecasts of wind generation for the next 48 hours (WINDFORFUELHH), followed by daily maximum wind forecasts for each day from 2–14 days (FOU2T14D). I merge these sources into a single 7 day hourly series, making certain assumptions about the hour in which the daily maximum will occur (for example, if the following day has a higher peak, I assume that the peak occurs at the end of the day).

I don’t have much sense of how good the wind generation forecasts from BMRS are, especially as BMRS do not offer historical forecasts. I have started storing the wind forecasts, so hopefully over time I can build up a better picture of this.

The wind generation forecasts are presumably based on the expected level of wind. However, weather forecasting agencies typically have additional data about how much higher or lower wind is likely to be, which can be used to give a sense of how much the wind forecasts are likely to change. Ideally I would take this data into account, however I haven’t yet found any open sources of this data.

Net Demand

One we have forecast gross demand, and solar and wind generation, we can calculate forecast net demand. Forecasts of these for the period from 15–21 July, as of 9am on 14 July, are shown in the chart below:

Estimating electricity prices

Net demand needs to be supplied by dispatchable generation (nuclear and fossil fuels), and the more dispatchable generation that is required, the higher resultant prices tend to be.

My approach has been to assume that the dispatchable generation supply curve (how much can be produced at what cost) over the next week will be similar to that over the past 14 days. I therefore estimate the curve of best fit, and apply it to net demand forecasts for the next 7 days. This assumption isn’t perfect: it may be that the likely available generation supply in the next week is different to the past two weeks. An analyst who could follow generator availability could adjust for this and improve their forecasts, however I don’t have the data for this.

Ideally we would fit day ahead electricity prices to day ahead estimates of net demand. Unfortunately BMRS do not provide historical data for day ahead demand or wind forecasts or wind forecasts. As a result, I have used the latest demand (FORDAYDEM, DANF) and wind generation (WINDFORFUELHH). I have compared the hourly net demand from these sources to day ahead hourly prices from Nordpool (https://www.nordpoolgroup.com/historical-market-data/).

If I look at the net demand and day ahead prices from 27 June to 10 July 2020, I get the scatter in the plot below. It is represented quite well (r=0.89) with a logarithmic function Price = 18.54 x ln(net demand)-150.6.

Every day I perform a regression and record the slope, intercept and r value. I then use the latest slope and intercept to convert my forecast net demand to prices:

Each day I store my forecasts, and the latest ones can always be seen at https://energy.guylipman.com/forecasts. You can see earlier forecasts by adding a before flag, eg https://energy.guylipman.com/forecasts?before=2020-07-11T23:00 (though I don’t have any before the evening of 11 July).

The motivation for this exercise was to generate forecasts of Octopus Agile tariffs, which are based on wholesale prices. The formula to convert from wholesale prices (in £/MWh) to retail tariffs (in p/kwh) is slightly different in each region. To save doing the conversions, the webpage can convert to any region, for example, for London, go to https://energy.guylipman.com/forecasts?region=C. (The region codes for each region are shown on https://en.wikipedia.org/wiki/Meter_Point_Administration_Number#Distributor_ID).

Further Work

The three main things I know I need to work on over the coming months are adjusting for daylight savings (never trivial when dealing with different data sources), adjusting for the impact of temperature on demand in winter (using temperature forecasts), and improving how I adjust for solar forecasts. If I do find an open source of uncertainty for wind generation I will certainly take that into account.

Disclaimer

Feel free to look at these forecasts, and use any of my ideas or code to try out different things. Hopefully you will find them interesting, but please don’t trade on the back of them, or even consider them to be reliable — as the Danish saying goes: “It is difficult to make predictions, especially about the future.”.