Displaying Energimolnet data to the user

Today I wanted to discuss how we work with timezones in our client applications. You may have noticed that we’ve discussed timezones here before; I wanted to write about it again because it is really the only slightly complex part of working with our API.

The confusing aspect is that the Swedish electricity industry assumes all meters are in GMT+1 (normal or winter time in Sweden) throughout the year.  That means that your bills, the data from meters and even Energimolnet’s data is skewed during summer time.

Summer time vs. normal time

The image below shows how we receive data during winter/normal time. The yellow is the window of time that makes up an electricity industry day. Whilst the purple is a user’s day. Quite sensibly the user’s day starts at 00.00 and ends just before 00.00 the following day.

time-stuff-normal

The two windows match perfectly and sometime during the following day we receive data for all 24 hours for the previous day.

However, during summer time this is how the windows look. The user’s day still starts at ends at 00.00 but that 00.00 doesn’t align with the electricity industry’s 00.00.

time-stuff-summer

The user’s day starts an hour earlier than the electricity industry’s.

That means that when data is sent from the distribution company to Energimolnet the next day we receive an hour of the current day as well (we always receive the yellow window). It’s not just an issue because we would have a strange graph with one hour of the current day, the more significant problem is that if we were to sum up the consumption for the day we must remember to sum up the consumption for the electricity industry day, and not the normal day, otherwise our apps would never match the bill one receives from the energy company.

So it’s key to remember that when you request a day from our API what you will receive is the electricity industry day, not the user’s day.

If you’re wondering what to display to the user we think it’s best to display:

  • All of the 24 hours you have received in an energimolnet request
  • The time in the USER’s timezone
  • During summer shift a displayed day by 1 hour so the user’s and the data’s windows are aligned

Displaying in the user’s timezone is slightly unintuitive, we’ve just said that we don’t get data in the user’s timezone. However, we wouldn’t want users to see their consumption peak according to us at 18.00 if they started cooking their dinner at 19.00. It wouldn’t make sense to them and it would defeat the purpose of the app.

It does mean that if their phone is in a different timezone (e.g., they’ve gone to the US) then the consumption will look strange but that hasn’t been too much of an issue yet. The best solution to that problem is to use the timezone of the location of the facility. For instance we could assume at the moment that all facilities are in Europe/Stockholm but supporting travelling users hasn’t been a priority to date.

The switch between summer and normal time

As this weekend we experienced the day where we switch from summer time to normal time I thought I would post pictures of how our apps handle that switch. You can see that on Saturday the displayed day starts with the hour 01.00-02.00 (as we explained above) and finishes with 00.00-01.00 of what the user’s Sunday.

In the displayed Sunday however we experience 02.00-03.00 twice which is why there is an extra hour between 01.00 and 06.00 on Sunday. The day finishes with the hour 23.00-24.00 which means that we’re back in alignment with the electricity industry day.

The Monday (Yesterday in the image) is normal time and matches perfectly with the electricity industry so the displayed day starts with the hour 00.00-01.00 and finishes with the hour 23.00-24.00

screenshot-1383056222279 screenshot-1383056232786 screenshot-1383056239596

To simplify things a little I made this little table:

+--------+----------------------+----------------------+--------------------+
|        |                      | displayed day starts | displayed day ends |
|        |                      | with hour            | with hour          | 
+--------+----------------------+----------------------+--------------------+
| Summer | GMT +2 (Summer time) | 01.00-02.00          | 00.00-01.00        |
+--------+----------------------+----------------------+--------------------+
| Fall   | GMT +2 / +1          | 01.00-02.00          | 23.00-00.00        |
+--------+----------------------+----------------------+--------------------+
| Winter | GMT +1 (normal time) | 00.00-01.00          | 23.00-00.00        |
+--------+----------------------+----------------------+--------------------+
| Spring | GMT + 1 / +2         | 00.00-01.00          | 00.00-01.00        |
+--------+----------------------+----------------------+--------------------+
| Summer | GMT +2               | 01.00-02.00          | 00.00-01.00        |
+--------+----------------------+----------------------+--------------------+
Advertisements

Testing Energimolnet’s API

What you’ll need:

  • A user account at Energimolnet (with facilities)
  • A developer account for Energimolnet. i.e., Client ID and Client Secret
  • [optional] A refresh token from Energimolnet.

If you want to test how to use the Energimolnet API you should find your user account (the one with facilities) and then:

  1. Log in to Energimolnet with that user account https://app.energimolnet.se (remain logged in)
  2. Open a new tab and go to https://app.energimolnet.se/api/1.1/users/me/meters?opt_pretty=1
  3. Open http://app.energimolnet.se/api/1.1/Documentation in another tab
  4.  Click the link in the List Meter Points section that says something like (note: the access token will be different): http://app.energimolnet.se/api/1.1/users/me/meters?access_token=0e639611a263e61141893b9cbc4c5365b3402114&opt_pretty=1
  5. You should see same response in both tabs. It should look like this and list all of the facilities in the account you are logged in as:

The access token in the second request is simply a token that makes sure the developer is allowed to look at that specific user’s data. We’ve made it so that you can test any request when you are logged in without worrying about implementing the OAuth security protocol. Of course you can only ever access your own data with this technique but it makes it easy to see what the request should be when you’re implementing in your own software.

If you’ve been given a refresh token by Playback Energy then you can get started by following the instructions that begin with Access Token request (using a Refresh Token). You simply send a POST request to the URL listed and include your client id, your client secret and the refresh token we gave you. You will get back an access token that will work for 3600 seconds.

After that token is expired you will receive a 401 error whenever you try and request and you will need to POST for a new Access Token.

Energimolnet Android OAuth2 sample

This post is intended to help people familiar with developing for Android get started with integrating the Energimolnet API in an app. You can find the source code here at Github. It’s a complete project so all you’ll need is a functioning Android development environment and you should be able to get started. Download the code, build and get it running now.

You will need a login to Energimolnet, so if you haven’t already, sign up at http://www.energimolnet.se now.

Then look at the documentation here: http://app.energimolnet.se/api/1.1/Documentation

authentication

Authentication with Energimolnet is via OAuth2.0 as it’s quite clearly the industry standard. That doesn’t, however, mean that it’s easy and we subscribe to the API Joy belief that sometimes cheating at OAuth is necessary  So this sample project and blog post is really designed to get a developer past the initial hurdle of authenticating.

The general flow for OAuth2 can be roughly described like this:

  1. Request an authorisation code by directing a browser at a URL with your client id, client secret and a callback url.
  2. Wait for the user to approve your app on http://www.energimolnet.se. When they ‘approve’ your app you receive a callback. When it arrives get a (very short lived) authorisation code from the parameters in the callback.
  3. Request an access token and refresh token using your authorisation code.
  4. Make normal API requests that always include your access token as a parameter.
  5. When the access token expires (1 hour for energimolnet) refresh the token using your long lived refresh token and client id and secret.

And in our app that means you should see a flow like this:

Now let’s see what that means in terms of code!

If we look at the onCreate method of the MyActivity

The key thing we do here is create an EnergimolnetAPI object and then get Authorisation URL and point the inbuilt browser at that URL. The difference between an Android app and a web based app is that the callback URL for the authorisation request can actually be any form we want. For instance here I’ve used: oauth://energimolnet The actual URL doesn’t really matter, what is important is that this activity registers that it wants to filter for this URL in the android manifest. If we look at the manifest:

The android:scheme and android:host properties are the important ones here. They tell the os that if we encounter a URL of the form oauth://energimolnet then direct that to our Activity, not to the browser. It’s important that the filter for BROWSABLE is also set as otherwise the os won’t know we can open URLs at all.

So when the user clicks Login  in this app and launches the browser they are taken to a browser to log in to Energimolnet. They are then asked whether they want to approve the app (demoapp) to access their data. If they click yes then the energimolnet server sends a redirect via the browser to that oauth://energimolnet URL we set as the callback. This URL is sent to MyActivity.

When MyActivity starts we check to see whether we’ve got the callback and if we have we know we can extract the authorisation code from that URL and then starts the task which actually requests the access token.

EnergimolnetAPI isn’t very complex if you’ve worked with network access in Android, if you haven’t then take some time and I suggest you read this Stackoverflow post.

This code simply POSTS a set of pairs to the Energimolnet server. Those pairs are the authorisation code, client id, client secret and the grant type (whether we’re refreshing a token or authorising for the first time).

That’s it! We’ve got an access token and a refresh token!

The Get Meters button in the Activity is now enabled and we can request some data from the API. That is handled by getMeters and then energimolnetRequest(URL).

The energimolnetRequest function adds an access token to a request and then places a GET request to an endpoint. If we get a 400 type error indicating that the access token has expired (or was never valid) then it tries to get a new token using our refresh token.

Done. We’re now requesting access tokens, refreshing tokens and looking up energimolnet API endpoints! Enjoy!

What’s the time?

According to the computer I’m writing on, right now it’s 15.52.

According to my phone it’s 15.52.

According to my laptop it’s 15.52.

According to the energy industry it’s 14.52.

The Swedish electricity industry decided that instead of following the very important Law 3 they would, of course, go their own way and store all time in their systems in GMT+1.

Law #3: Store it in UTC

This one is really more of a systems design law in general. I’ve lost count of how many times I have seen systems built with the timezone of the original company headquarters. Inevitably, unless you are quite disciplined, users will end up seeing their dates in your companies’ timezone. This is pretty annoying, especially if they are located on the other side of the globe.

Source: The 5 Laws of API dates and times

The issue is that all values are stored in GMT+1, not even in the local time of the facility. That means that during summer all times are off by 1 hour. If we displayed an electricity consumption chart for your office and your lunch break was between 12.00 and 13.00 the chart would be an hour wrong. It would always look like your lunch break started at 13.00. Which, if you were a company trying to save money on its consumption might be quite confusing. You can imagine the poor lonely employee who always eats his lunch later than everyone else getting blamed for using the microwave too much!!

What are Energimolnet doing to help?!

1.0

In the first version of our API (1.0) we had tried to conform to the rest of the world and use unix timestamps (UTC) when requesting a date and time. For instance you could request from 1366149600 to 1366236000 to request data for the 24 hours of the 17th of April and that is what you would receive. But our system had to do some very strange things in the background. For instance when we received data for 1 energy company day we would need to place that over 2 normal person days in our system. (Because they were sending 1am to 1am)

That was just the tip of the iceberg in terms of the complexities it introduced. It turns out it is very complex to turn non-UTC time into UTC time!

1.1

Our second attempt recognised that how we’d like our API to work, what seems logical, isn’t as important as understanding the industry we are a part of. But we also realised that making or API users work with millisecond resolution timestamps was a little ridiculous when our data was only ever 1 hour resolution and the most common call to the API was for a single day’s worth of values.

We think we’ve created a much simpler solution and that is that requests just ask for a date (no time) and we return all of the values that make up that time. For instance, a request for 201304 will return the 30 days of April, a request for 20130417 will return the 24 hours of the 16th of April. Easy to read, easy to debug, easy to understand!

However, there’s a trick! During summer we return the 24 hours that the grid company has sent us. And that means that we return from 01.00 to 01.00 local time and not 00.00 to 00.00. Of course, in winter we return 00.00 to 00.00. The reason is because if we returned 00.00 to 00.00 during the summer then the sum between the total for the day in our data and the total that the customer would see on their bill would be different, and the last thing we want is angry customers ringing their electricity companies up to complain about them being overcharged compared to what the Energimolnet service says or complaining to us that our numbers don’t add up.

TL;DR

Times aren’t stored logically within the energy industry. Ideally everything would be in UTC but it isn’t. Subsequently, we’ve moved away from UTC too. Rules were meant to be broken.