Displaying Energimolnet data to the user

Today I wanted to discuss how we work with timezones in our client applications. You may have noticed that we’ve discussed timezones here before; I wanted to write about it again because it is really the only slightly complex part of working with our API.

The confusing aspect is that the Swedish electricity industry assumes all meters are in GMT+1 (normal or winter time in Sweden) throughout the year.  That means that your bills, the data from meters and even Energimolnet’s data is skewed during summer time.

Summer time vs. normal time

The image below shows how we receive data during winter/normal time. The yellow is the window of time that makes up an electricity industry day. Whilst the purple is a user’s day. Quite sensibly the user’s day starts at 00.00 and ends just before 00.00 the following day.

time-stuff-normal

The two windows match perfectly and sometime during the following day we receive data for all 24 hours for the previous day.

However, during summer time this is how the windows look. The user’s day still starts at ends at 00.00 but that 00.00 doesn’t align with the electricity industry’s 00.00.

time-stuff-summer

The user’s day starts an hour earlier than the electricity industry’s.

That means that when data is sent from the distribution company to Energimolnet the next day we receive an hour of the current day as well (we always receive the yellow window). It’s not just an issue because we would have a strange graph with one hour of the current day, the more significant problem is that if we were to sum up the consumption for the day we must remember to sum up the consumption for the electricity industry day, and not the normal day, otherwise our apps would never match the bill one receives from the energy company.

So it’s key to remember that when you request a day from our API what you will receive is the electricity industry day, not the user’s day.

If you’re wondering what to display to the user we think it’s best to display:

  • All of the 24 hours you have received in an energimolnet request
  • The time in the USER’s timezone
  • During summer shift a displayed day by 1 hour so the user’s and the data’s windows are aligned

Displaying in the user’s timezone is slightly unintuitive, we’ve just said that we don’t get data in the user’s timezone. However, we wouldn’t want users to see their consumption peak according to us at 18.00 if they started cooking their dinner at 19.00. It wouldn’t make sense to them and it would defeat the purpose of the app.

It does mean that if their phone is in a different timezone (e.g., they’ve gone to the US) then the consumption will look strange but that hasn’t been too much of an issue yet. The best solution to that problem is to use the timezone of the location of the facility. For instance we could assume at the moment that all facilities are in Europe/Stockholm but supporting travelling users hasn’t been a priority to date.

The switch between summer and normal time

As this weekend we experienced the day where we switch from summer time to normal time I thought I would post pictures of how our apps handle that switch. You can see that on Saturday the displayed day starts with the hour 01.00-02.00 (as we explained above) and finishes with 00.00-01.00 of what the user’s Sunday.

In the displayed Sunday however we experience 02.00-03.00 twice which is why there is an extra hour between 01.00 and 06.00 on Sunday. The day finishes with the hour 23.00-24.00 which means that we’re back in alignment with the electricity industry day.

The Monday (Yesterday in the image) is normal time and matches perfectly with the electricity industry so the displayed day starts with the hour 00.00-01.00 and finishes with the hour 23.00-24.00

screenshot-1383056222279 screenshot-1383056232786 screenshot-1383056239596

To simplify things a little I made this little table:

+--------+----------------------+----------------------+--------------------+
|        |                      | displayed day starts | displayed day ends |
|        |                      | with hour            | with hour          | 
+--------+----------------------+----------------------+--------------------+
| Summer | GMT +2 (Summer time) | 01.00-02.00          | 00.00-01.00        |
+--------+----------------------+----------------------+--------------------+
| Fall   | GMT +2 / +1          | 01.00-02.00          | 23.00-00.00        |
+--------+----------------------+----------------------+--------------------+
| Winter | GMT +1 (normal time) | 00.00-01.00          | 23.00-00.00        |
+--------+----------------------+----------------------+--------------------+
| Spring | GMT + 1 / +2         | 00.00-01.00          | 00.00-01.00        |
+--------+----------------------+----------------------+--------------------+
| Summer | GMT +2               | 01.00-02.00          | 00.00-01.00        |
+--------+----------------------+----------------------+--------------------+

Where is the data?

When using Energimolnet’s API you will eventually experience that some values are missing. This is something every grid company deals with every day. One might think: How hard can it be? We agree with that thought!

But the reality is that data is missing every now and then. There are endless explanations as to why, but I’ll save you that information for now. Instead I want to explain how we deal with it and how that effects you as a developer.

If you get a value – it is correct

Our golden rule is: We only give you accurate values. You will never have to worry about incorrect values.

We provide three types of values: monthly, daily and hourly. In the ideal world the sum some 24 hour values always equal 1 day value. However, sometimes one hour value is missing and then we cannot aggregate them into a day value. That would not be accurate. Nor can we aggregate them to a month value.

So only relying on hourly volumes to aggregate a day or month value is not a good strategy. Therefore we import parallel feeds of day and month values from the energy company. This means that if one hour is missing, a parallel feed can provide data for the corresponding day or month value. The strategy give you the best data availability possible, regardless of missing underlying values.

If data is missing, try again

Usually Energimolnet can provide data the day after. But this is not a 100% truth. We rely on data from the energy companies. And there are some rules that they have to follow, these are stipulated by the government.

The regulation says that the energy company must have an accurate value within 5 business days. So month value can be expected the 5-7th day in a month. For hourly and daily values it is after 5-7 days.

I wish I could say that all energy companies comply with the regulation, but I can’t. Some companies cannot provide accurate data until the 11th in a month.

Important! Some meters are only measured by the month, and there for you cannot expect hourly or daily values. However, we try hard to make hourly volumes available for everyone.

That’s it

That’s it for now. If anything is unclear, don’t hestiate to contact me at luttkens@playbackenergy.se

Testing Energimolnet’s API

What you’ll need:

  • A user account at Energimolnet (with facilities)
  • A developer account for Energimolnet. i.e., Client ID and Client Secret
  • [optional] A refresh token from Energimolnet.

If you want to test how to use the Energimolnet API you should find your user account (the one with facilities) and then:

  1. Log in to Energimolnet with that user account https://app.energimolnet.se (remain logged in)
  2. Open a new tab and go to https://app.energimolnet.se/api/1.1/users/me/meters?opt_pretty=1
  3. Open http://app.energimolnet.se/api/1.1/Documentation in another tab
  4.  Click the link in the List Meter Points section that says something like (note: the access token will be different): http://app.energimolnet.se/api/1.1/users/me/meters?access_token=0e639611a263e61141893b9cbc4c5365b3402114&opt_pretty=1
  5. You should see same response in both tabs. It should look like this and list all of the facilities in the account you are logged in as:

The access token in the second request is simply a token that makes sure the developer is allowed to look at that specific user’s data. We’ve made it so that you can test any request when you are logged in without worrying about implementing the OAuth security protocol. Of course you can only ever access your own data with this technique but it makes it easy to see what the request should be when you’re implementing in your own software.

If you’ve been given a refresh token by Playback Energy then you can get started by following the instructions that begin with Access Token request (using a Refresh Token). You simply send a POST request to the URL listed and include your client id, your client secret and the refresh token we gave you. You will get back an access token that will work for 3600 seconds.

After that token is expired you will receive a 401 error whenever you try and request and you will need to POST for a new Access Token.

Energimolnet Android OAuth2 sample

This post is intended to help people familiar with developing for Android get started with integrating the Energimolnet API in an app. You can find the source code here at Github. It’s a complete project so all you’ll need is a functioning Android development environment and you should be able to get started. Download the code, build and get it running now.

You will need a login to Energimolnet, so if you haven’t already, sign up at http://www.energimolnet.se now.

Then look at the documentation here: http://app.energimolnet.se/api/1.1/Documentation

authentication

Authentication with Energimolnet is via OAuth2.0 as it’s quite clearly the industry standard. That doesn’t, however, mean that it’s easy and we subscribe to the API Joy belief that sometimes cheating at OAuth is necessary  So this sample project and blog post is really designed to get a developer past the initial hurdle of authenticating.

The general flow for OAuth2 can be roughly described like this:

  1. Request an authorisation code by directing a browser at a URL with your client id, client secret and a callback url.
  2. Wait for the user to approve your app on http://www.energimolnet.se. When they ‘approve’ your app you receive a callback. When it arrives get a (very short lived) authorisation code from the parameters in the callback.
  3. Request an access token and refresh token using your authorisation code.
  4. Make normal API requests that always include your access token as a parameter.
  5. When the access token expires (1 hour for energimolnet) refresh the token using your long lived refresh token and client id and secret.

And in our app that means you should see a flow like this:

Now let’s see what that means in terms of code!

If we look at the onCreate method of the MyActivity

The key thing we do here is create an EnergimolnetAPI object and then get Authorisation URL and point the inbuilt browser at that URL. The difference between an Android app and a web based app is that the callback URL for the authorisation request can actually be any form we want. For instance here I’ve used: oauth://energimolnet The actual URL doesn’t really matter, what is important is that this activity registers that it wants to filter for this URL in the android manifest. If we look at the manifest:

The android:scheme and android:host properties are the important ones here. They tell the os that if we encounter a URL of the form oauth://energimolnet then direct that to our Activity, not to the browser. It’s important that the filter for BROWSABLE is also set as otherwise the os won’t know we can open URLs at all.

So when the user clicks Login  in this app and launches the browser they are taken to a browser to log in to Energimolnet. They are then asked whether they want to approve the app (demoapp) to access their data. If they click yes then the energimolnet server sends a redirect via the browser to that oauth://energimolnet URL we set as the callback. This URL is sent to MyActivity.

When MyActivity starts we check to see whether we’ve got the callback and if we have we know we can extract the authorisation code from that URL and then starts the task which actually requests the access token.

EnergimolnetAPI isn’t very complex if you’ve worked with network access in Android, if you haven’t then take some time and I suggest you read this Stackoverflow post.

This code simply POSTS a set of pairs to the Energimolnet server. Those pairs are the authorisation code, client id, client secret and the grant type (whether we’re refreshing a token or authorising for the first time).

That’s it! We’ve got an access token and a refresh token!

The Get Meters button in the Activity is now enabled and we can request some data from the API. That is handled by getMeters and then energimolnetRequest(URL).

The energimolnetRequest function adds an access token to a request and then places a GET request to an endpoint. If we get a 400 type error indicating that the access token has expired (or was never valid) then it tries to get a new token using our refresh token.

Done. We’re now requesting access tokens, refreshing tokens and looking up energimolnet API endpoints! Enjoy!

What’s the time?

According to the computer I’m writing on, right now it’s 15.52.

According to my phone it’s 15.52.

According to my laptop it’s 15.52.

According to the energy industry it’s 14.52.

The Swedish electricity industry decided that instead of following the very important Law 3 they would, of course, go their own way and store all time in their systems in GMT+1.

Law #3: Store it in UTC

This one is really more of a systems design law in general. I’ve lost count of how many times I have seen systems built with the timezone of the original company headquarters. Inevitably, unless you are quite disciplined, users will end up seeing their dates in your companies’ timezone. This is pretty annoying, especially if they are located on the other side of the globe.

Source: The 5 Laws of API dates and times

The issue is that all values are stored in GMT+1, not even in the local time of the facility. That means that during summer all times are off by 1 hour. If we displayed an electricity consumption chart for your office and your lunch break was between 12.00 and 13.00 the chart would be an hour wrong. It would always look like your lunch break started at 13.00. Which, if you were a company trying to save money on its consumption might be quite confusing. You can imagine the poor lonely employee who always eats his lunch later than everyone else getting blamed for using the microwave too much!!

What are Energimolnet doing to help?!

1.0

In the first version of our API (1.0) we had tried to conform to the rest of the world and use unix timestamps (UTC) when requesting a date and time. For instance you could request from 1366149600 to 1366236000 to request data for the 24 hours of the 17th of April and that is what you would receive. But our system had to do some very strange things in the background. For instance when we received data for 1 energy company day we would need to place that over 2 normal person days in our system. (Because they were sending 1am to 1am)

That was just the tip of the iceberg in terms of the complexities it introduced. It turns out it is very complex to turn non-UTC time into UTC time!

1.1

Our second attempt recognised that how we’d like our API to work, what seems logical, isn’t as important as understanding the industry we are a part of. But we also realised that making or API users work with millisecond resolution timestamps was a little ridiculous when our data was only ever 1 hour resolution and the most common call to the API was for a single day’s worth of values.

We think we’ve created a much simpler solution and that is that requests just ask for a date (no time) and we return all of the values that make up that time. For instance, a request for 201304 will return the 30 days of April, a request for 20130417 will return the 24 hours of the 16th of April. Easy to read, easy to debug, easy to understand!

However, there’s a trick! During summer we return the 24 hours that the grid company has sent us. And that means that we return from 01.00 to 01.00 local time and not 00.00 to 00.00. Of course, in winter we return 00.00 to 00.00. The reason is because if we returned 00.00 to 00.00 during the summer then the sum between the total for the day in our data and the total that the customer would see on their bill would be different, and the last thing we want is angry customers ringing their electricity companies up to complain about them being overcharged compared to what the Energimolnet service says or complaining to us that our numbers don’t add up.

TL;DR

Times aren’t stored logically within the energy industry. Ideally everything would be in UTC but it isn’t. Subsequently, we’ve moved away from UTC too. Rules were meant to be broken.

What are the energy services of the future?

This was the question we tried to solve with an eclectic group of companies at a workshop we held together with BRG on the 26th of March.

The concept of the workshop was to gather representatives from a range of industries to discuss the energy industry and to develop concrete suggestions of products or services that could be developed with the energy data that Energimolnet is making accessible.

Before we describe the workshop though I think we need to answer why we think we need engaging tools and services. We outlined two reasons in the workshop. The first is that we’re tired of energy being boring! We work in an industry where the perception is that not much changed between 1900 and 2000! We maintain that there is no reason that energy awareness needs to be boring. That is just the perception we’ve all developed.

There is of course another reason, and that is that we believe that user engagement and efficiency will drive changes to energy consumption behaviour. This will lead to consumers who are more aware of their impact on the society’s energy consumption and, hopefully, reduced or improved consumption habits.

Inside and outside the Energy Data Fortress

Of course, Energimolnet might provide the data but someone needs to make these engaging products and services and that is why we gathered this group. The workshop had representatives from Swedish distribution companies, IT consultants, companies making energy smart products today and companies that are aiming to in the future. We don’t believe that any single industry, especially the energy industry, will be able to develop all of the engaging tools we need. So when we planned the workshop we knew we had to mix people from inside and outside what we call the Energy Data Fortress. The fortress only contains energy companies and their current system providers (large IT firms) so it’s quite easy to find companies that are stuck outside.

Energidatafortet

Problems problems problems

As our aim was to develop concrete product or service suggestions the first thing we had to do was identify problems to be solved. The workshop had no difficulty in generating 10s of problems that, in some way or another, relate to energy data. Some examples of the problems identified:

  • How can energy companies be incentivised to make data available?
  • What happens when electric cars are more common and everyone starts rapid-charging when they get home? (high power demands)
  • How do we differentiate between the consumption of different family members?
  • How do we differentiate between the consumption of different appliances?
  • Who can handle the data that distribution companies need to predict/respond to changes in usage?
  • How do you engage with customers when many of them don’t want to interact with their energy company?
  • How do you know if your consumption is appropriate? How does it compare to your neighbours?

Solutions?

Problems alone aren’t very interesting so then the group worked at coming up with different solutions. The solutions ranged from smart appliances to new types of contracts with energy companies.

Services which cut-costs for everyone

One of the problems identified was that energy companies struggle to justify their work in opening up energy data now. It seems like a positive thing for a company to do but what are the financial rewards for their work? How can customers, developers and energy companies all benefit? We were very interested to see this problem identified in the workshop as it’s one we’ve heard quite regularly in our conversation with the industry. New products and services are generally seen as costs to the distribution companies, not opportunities to reduce costs. The key is obviously to create a product that reduces costs for the energy company at the same time as reducing consumer costs or increasing consumer engagement. One of the solutions proposed was a service which analysed consumption habits and weather data and then predicted any peaks in consumption. These are easy enough to anticipate as forecasts of snow and low temperatures are good indicators of high load. On predicting such a consumption peak the service could push messages out to subscribers to a distribution company warning them that prices will be high and that they should aim to minimise or shift their usage. Of course if the peak is then reduced the distribution company saves money over time as they don’t need to invest so much in new infrastructure. In this way the grid company’s costs for initially opening up the data are offset by the savings in their network.

During the Workshop we didn’t want to advertise ourselves too much but we have spent a lot of time thinking about this problem. That is why Energimolnet offers a customer portal (“mina sidor”) to view the consumption data we store and doesn’t just make data accessible in an API. The concept was that to encourage energy companies to engage with Energimolnet we would need to offer something that they paid for today. So we offer the companies a customer portal, almost for free. Of course they get the benefit of opening up the data in that, like mushrooms in autumn, products will be developed for their customers without any effort from the energy company.

Combine these two ideas together and you get energy companies receiving customer portal pages for free, their customers receiving products and services that they would never have received otherwise and even the potential for the energy company to reduce their own costs. This is exactly the kind of result we were hoping for from the day!

The Smart and achievable washing machine

We here a lot about smart appliances and even The Internet of Things in the media but it’s always “appliances of the future” that require some vital piece of equipment that actually very few houses have. The typical examples of this are smart appliances that turn off/down when power usage in a household is high. This is a great concept but requires some device to measure power usage and then make that data available in an open format to all of one’s appliances. This device doesn’t exist yet and is still years off according to the energy companies we trust. However, a concept of a smart washing machine was developed yesterday that didn’t require anything more than an internet connection. With an internet connection the device could compare the price of washing now versus the price of washing in an hour or two, it could sense if it wasn’t full and then determine how much a user could save if they waited until it was full, it could also suggest the best time to wash to even the household’s electricity load and thus avoid any high-power usage fees.

New energy contracts

Our energy company participants noted during the day that not everyone cares about their consumption, or at least not everyone cares about all of their consumption. For instance, I might care a great deal about our office’s consumption as we’ve got a large space with lots of computers and lighting but I don’t care about the consumption in my apartment as much because I know it’s tiny, I’m rarely home and I don’t own any high consuming appliances. How can an energy company know what level engagement they should aim to have with each customer? This lead to the idea that an energy company could develop plans where you select your level of engagement. For most customers all they need to know about is if their consumption is “all clear” and if not there needs to be some subsequent investigation. However, some customers are already interested in saving money and electricity and want to be engaged. These customers could be seen as a valuable resource where they provide feedback on the tools created, they provide an initial market of “early adopters” as they’re called in the product development world that can help companies that want to develop products and services. What an energy company should recognise is that they’re not the typical customer and giving them the same information as an uninterested consumer will not make either happy. Companies should recognise that, at least for the moment, there are levels of interest.

How do the tools become interesting?

A great point brought up during the meeting was that the RunKeeper app has made running times interesting and social. If you asked someone 20 yeas ago if the time it took them to run around the block was interesting they probably would have laughed. There are two points here, one is that competition can increase engagement to a level financial incentives would never reach. For instance, even if gasoline was twice as expensive as now people wouldn’t be sharing their car’s consumption on Facebook, it takes something more than just financial rewards to increase engagement. The other is that boring things can almost always be made interesting with the right angle and it’s everyone’s job to work out what that angle is.

Were we successful?

We left the workshop with some terrific solutions, some of which we’ve described here but all of which brought something new and interesting to the energy product world.

However, it is important to reflect on the fact that Energimolnet’s entire purpose is to make it easier for everyone to build products and services. We are focussing on opening up this data and making it accessible in the belief that the rest of the world has better product ideas than us, has better concepts than our workshop participants, and can see markets none of us would have anticipated. We were around 20 people gathered in Göteborg, Energimolnet will make it possible for the rest of the world to try solving these problems.

Thanks to our participants:

Ale Elförening | BRG | Exebia| GU | Knycer | Lerum Energi | Nindev | Playback Energy | SP | Varbergortens elkraft | WSP Systems

thanks also to Tillväxtverket, from which we received some financial support to hold the workshop.