Analysing UK electricity prices (Part 1)

Since I moved to a time-of-use tariff with Octopus Energy (as discussed in my recent post), I have been thinking a lot more about half hourly electricity prices, how much they vary, and to what extent they can be predicted. I have started analysing these, and thought it might be of interest to others.

Disclaimer: Please do not make significant decisions on the basis of this analysis. UK energy markets are changing rapidly, and the future is likely to differ significantly from the past. In addition, I am sure I have made mistakes in this analysis. Feel free to get in touch if you have any questions.

How the UK electricity market works

Before I get into the data, I will provide a short (simplified) explanation about how the UK electricity market works.

Electricity generators enter into contracts to sell electricity to retailers (who expect to use that electricity for their residential and commercial customers). These can be traded several years ahead, right through to the hour in question. In fact, it is not just generators selling to retailers: other participants enter into these forward contracts to buy and sell electricity, including on exchanges.

If we got to the period in question, and the amount of electricity generators supplied and retailers received, matched their contracts, all would be nice. The grid would be in balance (ignoring factors like congestion and transmission losses). The retailers would pay the generators the price they had agreed.

In practice, customers are likely to consume more or less electricity than the retailers predicted, and generators may choose (or be forced by outages) to produce less than they’ve sold in their forward contracts. When this happens, National Grid’s balancing mechanism kicks in, and they buy or sell electricity to ensure a balance. Then, another organisation, Elexon, are responsible for calculating an imbalance price for each half hour. Any participants that have used more than they had bought (or generated less than they had sold) have to pay the imbalance price for the amount they are short. Any participants that have used less than they had bought (or generated more than they had sold) receive the imbalance price for the amount they are long (have excess).

The imbalance price can be very volatile because it is expensive to store electricity. In 2018 it ranged from £-150/MWh (ie you had to pay if you produced too much electricity) to £990/MWh, so participants have a strong incentive to minimise their imbalances.

Sourcing Imbalance Prices from Elexon

The primary data I was interested in sourcing was the imbalance prices for each half hour. These are displayed on the Elexon website (https://www.bmreports.com/bmrs/?q=balancing/systemsellbuyprices). I note that they refer to buy prices and sell prices, though in fact the are almost always the same. However, Elexon also provide an API to load the data programmatically — I used python. You have to sign up to Elexon to get an APIKEY, but it is free. The following function can be used to extract a number of Elexon reports.

def load_data(report, date):
    import requests
    from io import StringIO
    import pandas as pd
    l1 = 'https://api.bmreports.com/BMRS/'
    l2 = report
    l3 = '/V1?APIKey=' + APIKEY + '&'
    l7 = 'ServiceType=csv'
    d = date.strftime('%Y-%m-%d')
    if report in ['DERSYSDATA']:
        l4 = "FromSettlementDate="+d+"&ToSettlementDate="+d+"&"
    elif report in ['FORDAYDEM']:
        l4 = "FromDate="+d+"&ToDate="+d+"&"
    else:
        l4 = ""
    r = requests.get(l1+l2+l3+l4+l7 )
    data = pd.read_csv(StringIO(r.text), 
                       header=None, 
                       skiprows=1)
    data = data.iloc[:-1]
    return data

This code requires APIKEY to be loaded in memory (you can get this when you create your Elexon profile). It returns a pandas dataframe, with 48 rows, one for each half hour. In report DERSYSDATA, column 3 contains the imbalance price.

An important note on UK electricity data: each day runs from 23:00 the previous day to 23:00 (UK prevailing time, ie either UTC or BST). As a result, the last Sunday in March will have 46 half-hours, and the last Sunday in October will have 50 half-hours. This is something that causes endless headache for developers/analysts.

Then, to get the imbalance price for each half hour in the year, I called the following code. It took about 3 minutes to run on my computer. (Rather than making 365 calls of one day at a time, I could have done 12 batches of 30 days, but it is simplest to show it this way):

startdate = pd.datetime(2018,1,1)
d1tables = []
 
for i in range(365):
    data1 = load_data('DERSYSDATA', 
                      startdate + pd.offsets.Day(i))
    d1tables += data1[3].tolist()

Day Ahead and Intraday Auctions

I mentioned in the introductory section on the UK electricity market that participants may enter into contracts up to the hour in question, which can be used to reduce their exposure to imbalance prices. To make this easier, a number of auctions are held allowing participants to buy or sell any volumes they want.

The main auction is held at 11am UK time, covering all the hours the following day (or more precisely, from 23:00 that night until 23:00 the following night, as noted above). This auction process takes bids and offers on both APX (part of EPEX Spot) and N2EX (a joint venture between Nordpool and Nasdaq), as well as bids and offers in interconnected markets (and interconnector capacities). It determines a single clearing price for each hour for the UK market, and all bids above that price and offers below that price will clear.

In addition, APX hold later auctions, at 3:30pm, at 5:30pm and at 8:00am UK time. These auctions are for individual half-hours, and have lower liquidity. The 3:30pm auction results are used by Octopus Energy as the basis for their time of use tariff.

Finally, N2EX and APX also offer continuous trading of individual half hours. Unfortunately these bids and offers, and any trades ultimately done, are not visible to the public.

Sourcing Auction Prices

I was able to source the 11am day ahead auction prices for 2018 from N2EX (https://www.nordpoolgroup.com/historical-market-data/) — look under N2EX Day Ahead Auction Prices, Hourly, GBP. Note that these are hourly, and I had to remove the row for the missing hour in March.

I wasn’t able to get any historical auction prices from APX — my understanding is that these are only available for purchase. The best I could do was take the Octopus Agile tariffs (available for download at https://octopus.energy/agile)/, and I know the formula to convert from the 3:30pm auction prices to get the Octopus tariffs. There are two challenges with this approach. Firstly, the Octopus tariffs have a cap of 35p/kWh, so for any half hours with that tariff, I don’t know the actual auction price. Thankfully this only occurred 59 half hours in the 2018. Secondly, on Christmas day, Octopus set the tariff to 0p/kWh from 11am-3pm, which means I don’t know what the auction prices were.

Analysing Prices

Firstly, I compared the average and standard deviation of the prices:

+--------------+--------+--------+--------+---------+
|              |  Mean  | StDev  |  Min   |   Max   |  
---------------+--------+--------+--------+---------+
| 11am Hrly    | 57.44  | 13.11  | 9.09   | 191.55  |
+--------------+--------+--------+--------+---------+
| 3:30pm HHrly | 57.29  | 13.75  | -11.0  | 166.65  |
+--------------+--------+--------+--------+---------+
| Imbalance    | 57.35  | 30.25  | -150.0 | 990.0   |
+--------------+--------+--------+--------+---------+

My first conclusion from this is that there is no significant overall bias in any of the prices, but the imbalance prices have much higher variability. This is as I expected, given that the imbalance prices offer the market much less opportunity for demand or supply response.

The next thing I looked at was the standard deviation of movements between the three prices, for different half hours. The movement from 11am Hrly to 3:30pm HHrly had a standard deviation of £5.28, and the movement from 3:30pm HHrly to the imbalance price had a standard deviation of £28.60. Another way of considering this is that the 11am Hrly and 3:30pm HHrly have a correlation of 92%, while the 3:30pm HHrly and the imbalance price have a correlation of 36%. In other words, knowing what the 3:30 auction prices are, tells you something about the imbalance prices, but there’s still a lot of uncertainty.

The final piece of analysis I wanted to do in this post ties in with some of my motivation for this analysis. As discussed in my previous post on time of use tariffs, most of my electricity usage for a boiler, which I use for between 30 and 120 minutes per day, usually in the cheapest half hours. As a result, I was curious how much cheaper the cheapest periods were if I was being priced off imbalance prices rather than the 3:30pm auction.

+-------------+---------+----------+----------+--------+ 
| Mean        | 30 min  |  60 min  |  120 min | 24 hrs |
+-------------+---------+----------+----------+--------+
| 3:3pm HHrly | 41.63   |  42.24   |  43.17   | 57.29  |  
+-------------+---------+----------+----------+--------+
| Imbalance   | 18.50   |  22.77   |  27.83   | 57.35  |
+-------------+---------+----------+----------+--------+
+-------------+---------+----------+----------+--------+ 
| StDev       | 30 min  |  60 min  |  120 min | 24 hrs |
+-------------+---------+----------+----------+--------+
| 3:3pm HHrly |  8.61   |   8.44   |   8.08   |  7.65  |  
+-------------+---------+----------+----------+--------+
| Imbalance   | 31.17   |  25.38   |  19.45   | 17.29  |
+-------------+---------+----------+----------+--------+

This shows that even though the average of the whole day was the same whether we used Imbalance or the 3:30pm Half Hourly prices, the cheapest 30–120 minutes were significantly cheaper when we used the imbalance prices. This must be balanced against the fact that they are so much more volatile (as shown by the standard deviation), but I would be inclined to accept that volatility to get the much lower prices.

An important assumption in this calculation is that I would be able to select the cheapest 30–120 minutes. With the 3:30pm Half Hourly auction, we know all the prices before the start of the 24 hours. With the Imbalance prices, we don’t even know them until after the hour. In order to optimise on Imbalance prices, we would need a way to develop better forecasts of what they were likely to be. I am keen to learn the extent to which that is achievable, and will be looking into that in a later post.

This post has benefitted from a number of conversations with current and former colleagues, Karolis Petruskevicius at www.homelyenergy.com, some of the people at Octopus Energy, and a course I went on at Elexon. All errors remain my own.