US Energy Information Administration has a new technique for making APIs accessible...

42 views
Skip to first unread message

Mark Boyd

unread,
Mar 19, 2015, 7:41:58 PM3/19/15
to us-govern...@googlegroups.com
Hi all

I had a great discussion with Mark Elbert from the U.S. Energy Information Administration today about the new Microsoft Excel add-on tool that they created with the Federal Reserve to enable energy and economic API data to be channelled directly into spreadsheets. As a writer for ProgrammableWeb, i am seeing this sort of strategy being used by some of the leaders in API adoption and uptake in the startup sector, so it is great to see the approach being applied to government APIs.

I have written up the details here:

But the source material was the press release and blog from EIA:

All the best
m.

Eric Mill

unread,
Mar 22, 2015, 7:02:09 PM3/22/15
to Mark Boyd, us-government-apis
I think this a helpful, user-centric feature for EIA to add, but I don't see why it's to be celebrated as an API success.

* It's a closed source release. The St. Louis Federal Reserve's original extension was too. It's weird to see EIA thank STL FR for generously sharing their code. It's taxpayer-funded code. I should be able to modify it myself for other government APIs.
* It's a native add-on for a proprietary commercial platform. Nothing wrong with doing that, but I don't find it a particularly inspiring use of APIs either. I'd be a bit more inspired if an outside party had, using publicly documented APIs, built it to solve their problems and released it to others.
* Maybe it uses the exact same API endpoints they offer to the public, and maybe it uses some private ones. I'm unable to look at the code to know for sure, nor do they seem to document it anywhere.

I don't mean to be all negative about a cool tool that EIA built for an important part of their userbase. They're to be commended for doing it. I also have generally really appreciated the EIA's commitment to making public APIs.

But I don't see where a gov agency releasing a closed-source tool for their own data, that may or may not use their public APIs, fits into the narrative of open APIs making government more transparent or reusable.

-- Eric

--
You received this message because you are subscribed to the Google Groups "US Government APIs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to us-government-a...@googlegroups.com.
To post to this group, send email to us-govern...@googlegroups.com.
Visit this group at http://groups.google.com/group/us-government-apis.



--

Mark Boyd

unread,
Mar 22, 2015, 7:43:24 PM3/22/15
to us-govern...@googlegroups.com, ma...@mgboyd.com
These are useful insights Eric. 

Certainly looking at this compared with, say, the Department of Labor's approach to sharing the source code for the tools they create based off their APIs was something i wanted to address in a followup article this week. 

Also, i was trying to (gently) flag the potential of making Google Docs tools as an alternative to this Windows operating system-only Excel tool, i could have been more forthright in addressing that in the article (and even in my initial interview). I'll go back to EIA and discuss the points you make and try and do a followup article this week.

m.
To unsubscribe from this group and stop receiving emails from it, send an email to us-government-apis+unsub...@googlegroups.com.

To post to this group, send email to us-govern...@googlegroups.com.
Visit this group at http://groups.google.com/group/us-government-apis.

David Berlind

unread,
Mar 24, 2015, 6:38:38 PM3/24/15
to Eric Mill, Mark Boyd, us-government-apis
Eric, also, in addition to what Mark said in his response, you are welcome to author a slightly more detailed (give specifics in case someone didn't read the original piece... what's proprietary, what's not, etc.) version of your post here for ProgrammableWeb. Let me know (and sorry, I won't be able to make the upcoming meetup next week).

Mark Boyd

unread,
Mar 30, 2015, 3:58:10 PM3/30/15
to us-govern...@googlegroups.com, er...@konklone.com, David Berlind
Following on from Eric's legitimate concerns about the EIA tool release, i did a followup interview with EIA and reached out to an API user as well (i also tried to speak with renewable energy non-profits about what they need to use energy information APIs, but no luck yet).

Here's the followup article:

Also on government transparency via APIs, we published an article this week discussing the lack of an API from the Securities and Exchange Commission, how startups are using SEC's RSS feeds to collate data instead, and SEC-alternative API providers, including one that was created by an ex-SEC employee who was so frustrated by the lack of an API from the SEC that he learnt how to code and built one himself:

Thanks Eric for raising the initial issues, i'm keen to strengthen and improve my reporting around government API actions, so this was a really valuable exercise for me and hopefully adds to the conversation.

m. 


On Tuesday, March 24, 2015 at 11:38:38 PM UTC+1, David Berlind wrote:
Eric, also, in addition to what Mark said in his response, you are welcome to author a slightly more detailed (give specifics in case someone didn't read the original piece... what's proprietary, what's not, etc.) version of your post here for ProgrammableWeb. Let me know (and sorry, I won't be able to make the upcoming meetup next week).
On Sun, Mar 22, 2015 at 7:01 PM, Eric Mill <er...@konklone.com> wrote:
I think this a helpful, user-centric feature for EIA to add, but I don't see why it's to be celebrated as an API success.

* It's a closed source release. The St. Louis Federal Reserve's original extension was too. It's weird to see EIA thank STL FR for generously sharing their code. It's taxpayer-funded code. I should be able to modify it myself for other government APIs.
* It's a native add-on for a proprietary commercial platform. Nothing wrong with doing that, but I don't find it a particularly inspiring use of APIs either. I'd be a bit more inspired if an outside party had, using publicly documented APIs, built it to solve their problems and released it to others.
* Maybe it uses the exact same API endpoints they offer to the public, and maybe it uses some private ones. I'm unable to look at the code to know for sure, nor do they seem to document it anywhere.

I don't mean to be all negative about a cool tool that EIA built for an important part of their userbase. They're to be commended for doing it. I also have generally really appreciated the EIA's commitment to making public APIs.

But I don't see where a gov agency releasing a closed-source tool for their own data, that may or may not use their public APIs, fits into the narrative of open APIs making government more transparent or reusable.

-- Eric
On Thu, Mar 19, 2015 at 7:41 PM, Mark Boyd <ma...@mgboyd.com> wrote:
Hi all

I had a great discussion with Mark Elbert from the U.S. Energy Information Administration today about the new Microsoft Excel add-on tool that they created with the Federal Reserve to enable energy and economic API data to be channelled directly into spreadsheets. As a writer for ProgrammableWeb, i am seeing this sort of strategy being used by some of the leaders in API adoption and uptake in the startup sector, so it is great to see the approach being applied to government APIs.

I have written up the details here:

But the source material was the press release and blog from EIA:

All the best
m.

--
You received this message because you are subscribed to the Google Groups "US Government APIs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to us-government-apis+unsub...@googlegroups.com.

To post to this group, send email to us-govern...@googlegroups.com.
Visit this group at http://groups.google.com/group/us-government-apis.

--
You received this message because you are subscribed to the Google Groups "US Government APIs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to us-government-apis+unsub...@googlegroups.com.

Eric Mill

unread,
Apr 2, 2015, 1:13:13 AM4/2/15
to Mark Boyd, us-government-apis, David Berlind
This is a great followup, Mark, thank you for covering the issue so thoroughly!

It's really interesting that the API key requirement factors into the comments of Professor Brigida, and the EIA itself. 

Prof. Brigida is right on that the API key requirement makes it difficult to do interactive JavaScript analyses without exposing the API key. This came up all the time (and I'm sure still does) at the Sunlight Foundation, which does a lot of JavaScript data visualizations that talk directly to its APIs, as do many of its clients. Typically, the answer is to just put the key in your source code, and if it does get stolen or abused, then worry about it on the enforcement end. Sunlight is pretty chill like that, and of course the API key is only for read-only public data stuff. 

The EIA's argument that the API key requirement is what's halting the publishing of their source code doesn't hold water, though. Not wanting to expose the API key may very well mean that when you download the extension, the extension needs to be a compiled, encrypted binary so that people can't yank the key out of it. However, they could absolutely still publish the source code online for people to view, fork, and improve -- and just not publish the API key in the released code.

While it's possible that kind of workflow might mean some slight refactor of the Excel plugin's code, avoiding hardcoding of keys/passwords into source code is an *extremely* common (and good) practice. Lots of open source projects require the use of private keys to communicate with authenticated systems. It's not a blocker to making the Excel plugin open source. 

For an example of this, see 18F's analytics-reporter, a command line tool for speaking with the Google Analytics API, that powers https://analytics.usa.gov. The tool requires a sensitive private key that has authorized access to the US government's Google Analytics account, and the contents of this key are provided by reference via an environment variable. Other software systems will have different appropriate methods of doing this kind of thing.

In any case, not suggesting you go back for another follow-up. :) This resolved some important questions (like whether or not the plugin uses only public methods), and got some interesting detail from Prof. Brigida. Glad to see this stuff getting attention!

-- Eric
Reply all
Reply to author
Forward
0 new messages