A website for viewing your smart meter data

This post explains the ideas behind a website I have built to allow UK owners of smart meters access to their data, in a way that makes that data useful to them. I hope that sharing my thinking with others in the field will generate some useful suggestions and debate.

Key principles

My goal is to build something that makes it free, and as easy as possible for as many people as possible to access their data.

Next, I know that a lot of customers aren’t very trusting when their data is concerned. I want to make it as easy as possible for people to trust the website, and to learn to appreciate the value in having access to their data. To do this, I will take data security concerns extremely seriously. And I will always make decisions in the customers’ interests (rather than, say, in the interests of companies that may want access to customer data).

Design decisions

I chose a website for a number of reasons (over, say, a mobile app). Most customers have access to a web browser. Web access allows use on a computer screen, which increases the real estate for me to provide information. I already know python (which can be used to build websites), reducing the learning curve. And websites offer a much quicker turnaround on changes and releasing new features, without any worry that customers will be stuck on old versions.

The next difficult decision was what customer data to store. I have previously built websites that don’t store any customer data, making them very slow, especially if you want to look back a year. The other extreme would be one in which customers logged in, and provided their login details for their utility. Some customers are happy to do this, but I worried that many would not, reducing the accessibility of the service. I also worried about the risks if any of this data got stolen.

I decided to store consumption data, but nothing else, and only for as long as needed. The user provides their login key to load the data, however the login key is not stored. The user can then perform any number of calculations on that data, before deleting it (or it is deleted automatically within a few days).

I have initially hosted my website on a Digital Ocean virtual server, with NGINX/Django/PostgreSQL. Even the cheapest tiers are good enough to get started, and handle hundreds of customers a day, and there is plenty of scope to scale up if or as I need. I will be adding additional features over the coming weeks, so it may occasionally be down, but it should be available most of the time. Obviously nothing would stop the logic being ported across to another technology stack should it be necessary.

A final design decision is that I mostly stick to stateless webpages, that is, what is shown on the webpage is largely dependent on the url. It may not look as tidy having lots of parameters in the url, and may not run as quickly. However, from the perspectives of consistency and transparency, it makes development much simpler, and also allows users to bookmark particular reports/configurations.

The Website

The core user flow is:

1. Go onto the website and provide a key to access your consumption data. At this point either Octopus or n3rgy are available, but I am eager to add more.
2. Choose to load the data (this takes 10–20 seconds if you’ve got 18 months of data). I don’t currently allow storing data before January 2019.
3. I do have the ability for customers with previously loaded data to look for new data and ‘update’.
4. Access website functionality based on your data: view charts of consumption, cost and carbon intensity, calculate bills between two dates, export data to csv. This includes gas consumption and electricity export if you have these. Costs can be based on fixed tariffs or time-varying ones (at the moment I have a number of Octopus time-varying tariffs like Agile and Go available, but I can easily add more).
5. Delete your data from storage. If you don’t delete it, it will be deleted automatically within 7 days.

If you don’t want to load data, you can also try out the website functionality using a made-up data set.

Data Structure

Feel free to ignore this section, but I figured it would be useful/interesting for some readers.

I have three main tables in which I store data:

• sm_quantities which stores the quantity consumed for each half hour for each user, for each of electricity consumption, gas consumption and electricity export.
• sm_hh_variable_vals, which stores values for each half-hourly variable. Currently I have a variable for each of Octopus’s half-hourly tariffs for each region, one for carbon intensity, and one for Elexon’s domestic deemed profile.
• sm_d_variable_vals, which stores values for each daily variable. Currently I only have a variable for Octopus’s gas tracker product for each region.

I have a number of daily scripts that keep the variable_vals tables populated with the latest prices and carbon intensities.

My website is able to perform all sorts of calculations efficiently using sql, using the data in these tables.

Call for Suggestions

I’d love to get suggestions, especially from others that have thought about this topic before. A few particular points I’d love to hear on.

• Are there any UK utilities other than Octopus that offer customers with smart meters API access to their data? I’d be very happy to write the interface layer to load that data into the database.
• What are the priorities for other time-varying tariffs that I should include? I would like to include eco7, but would most customers know what time their ‘night’ period covers?
• Are there other charts or reports that would be useful for customers?
• I’d like to put the code into github, and make it open source. I’d also like to have other people able to design reports. Is this something that others would find useful? Is anyone able to offer guidance on how best to do this?
• Are there organisations that I should be involving in this project?
• Does anyone know if this kind of thing is being done in other countries?

Feel free to comment below, or reach out to me directly via twitter or email.

If you’ve made it this far, you will probably want to visit the website itself on https://energy.guylipman.com/sm/home, otherwise I’ve included some screenshots below.