Tackling API Discovery In The Federal Government

[Kin Lane]

Happy weekend everyone,

I wanted to share some of my recent work with the group, around trying to tackle API discovery in the federal government. I'm evolving on my work with /data.json, more recently my APIs.json project, and exploring how we can stimulate API discovery with API search engines like APIs.io, and API search catalogs like ProgrammableWeb, and Mashape, across government.

To prepare myself, I looked at all 77 of the federal agency developer areas, with the mission of identifying the low hanging fruit that I could target as part of preliminary work to generate APIs.json for federal government agencies, for inclusion in distributed API search engines and directories, including, but not limited to APIs.io. 

I chose to kick off my work by targeting the federal agency domain API at explore.data.gov, the federal agency directory API, social media registry API, and the mobile app gallery API at www.usa.gov.

My goal was to create machine readable (swagger) specs for all 4 APIs, and an APIs.json to demonstrate, one potential way that we can tackle discovery across federal agencies--using explore.data.gov and www.usa.gov as guinea pigs. Since there were two separate domains, two separate APIs.json files were needed, with a total of four APIs:

• www.usa.gov/apis.json
  
• explore.data.gov/apis.json
  

As it stands, I can submit both of these APIs.json files to an API search engine like APIs.io. However I'm not the authoritative source, so it will index, but not list me as authority for domain. Ideally each federal agency would adopt the APIs.json, bring internally, published within their domain, and maintain as APIs come and go within the domain.

My goal is to generate APIs.json for all federal agency domains, and underlying machine readable API definitions for their API inventory. I have 120 of the 190 APIs targeted, that I evaluated while looking through the 77 federal agency developer areas. The process is time consuming, so it won't happen overnight--any help is appreciated.

Once we have machine readable API definitions for all current APIs in the federal government, and APIs.json file for every government domain, I think we'll have a new vantage point to look at APIs in government from--allowing for a renewed conversation about how we find valuable API resources across the federal government.

I'm always looking for feedback on my work via my blogs, each of the research sites I've setup on Github for www.usa.gov/apis.json and explore.data.gov/apis.json (each domain I do will have its own repo), or directly via email--your thoughts are so valuable.

[Philip Ashlock - XI]

Thanks Kin. Since agencies are already required to document their data with the data.json schema and that includes documenting APIs I think it might be easier to take the incremental steps to use that existing momentum and mandate rather than try to bootstrap something entirely new. Obviously whatever structured form we can capture this in, it should be trivial to do transformations to another format. It's just that there's already a process in place to capture this with data.json and that process includes improving the data.json schema to accommodate use cases like this. In fact, next week there will be a summit to hash out remaining issues like this. Perhaps you can reference the work you've done on the this issue thread: https://github.com/project-open-data/project-open-data.github.io/issues/291

--

Philip Ashlock
Chief Architect, Data.gov
U.S. General Services Administration

[Gray Brooks]

Kin - thanks a bunch - this is awesome.  I'm going through and will follow up with more thoughts, but thanks for looking at this sector in such detail.  

I definitely agree with adding to the discussion in that issue and think that these norms will reconcile well.  My goal is to see govt's APIs represented both thoroughly and efficiently.  I need to try and follow up to see all of these APIs represented in their agency's public data listings (data.json files) but also want to make sure that those APIs are best bubbling up into apis.io, mashape, programmableweb, etc. 

Digging into this more and will follow up.  Thanks again, Kin.  

Gray

------------------------------

c - 205.541.2245

Sr. API Strategist

Team 18F - GSA

* /Developer Program

* US Government APIs listserve

[Gray Brooks]

I'll be working on this more and will try to follow up soon, but in the meantime, I really want to encourage every govie to check out her or his APIs in Kin's blogpost.  There's a bunch of really solid, important feedback that Kin gives as he went through, looking at pretty much every one of our APIs.  

I strongly encourage all of us to take his feedback in that post to heart!  

Gray

[Kin Lane]

Thank Phil, for the link to that conversation.  I will definitely share what I've been up to, coming from private sector, on that Github thread.. 

I also will also be considering better linkage with API references in data.json, and back again.

Lots to think about when data has a supporting API, when APIs have related data, or also when agency API is compute resource vs. data. 

My goal is to bring what is being done in private sector, share with gov, then find best way to augment, extend with existing gov efforts.

[Maher, Mary - ERS]

Kin Lane, you ROCK! I’ve finally gotten around to your blog post, which is a really great review of the gov APIs—even though you had mixed feelings on ours ;)

 

This is a really useful roundup, and provides lots of good lessons and advice. We’ll certainly take it to heart.

 

 

Mary Maher

Chief, Web Services Branch

Information Services Division

USDA’s Economic Research Service

202.694.5126   |   mem...@ers.usda.gov 

This electronic message contains information generated by the USDA solely for the intended recipients. Any unauthorized interception of this message or the use or disclosure of the information it contains may violate the law and subject the violator to civil or criminal penalties. If you believe you have received this message in error, please notify the sender and delete the email immediately.

[Gray Brooks]

Kin - you touched on some really solid points about public feedback of government APIs.  That, coupled with Navin's very fruitful request for feedback motivated me to bring together the list of government APIs that have a public GitHub issue tracker set up for them (also pasted below).  As you can see from the feedback being offered up for the SAM API, there's a lot of value for all involved to queue up plenty of feedback for these agencies to work on.  

Kin - any interest in parsing out some of your feedback for USDA in their public issue tracker so that they can better process it?   

And everyone else, if you're game - take a few minutes and give an agency a some TLC so that they can better improve their APIs.  :)

Gray 

Department of Agriculture

• All APIs / Issue Tracker

Department of Health and Human Services

• FDA Drug Adverse Events API / Issue Tracker
• FDA Device Recalls API / Issue Tracker
• FDA Food Recalls API / Issue Tracker

Department of Labor

• All APIs / Issue Tracker

Department of State

• Office of the Historian API / Issue Tracker

Consumer Financial Protection Bureau

• HMDA API / Issue Tracker

General Services Administration

• FBOpen API / Issue Tracker
• SAM API / Issue Tracker
• All APIs / Issue Tracker

National Aeronautics and Space Administration

• All APIs / Issue Tracker

US Agency for International Development

• All APIs / Issue Tracker

[Maher, Mary - ERS]

 

 

Mary Maher

Chief, Web Services Branch

Information Services Division

USDA’s Economic Research Service

202.694.5126   |   mem...@ers.usda.gov 

 

From: us-govern...@googlegroups.com [mailto:us-govern...@googlegroups.com] On Behalf Of Gray Brooks
Sent: Wednesday, July 23, 2014 6:03 PM
To: us-govern...@googlegroups.com
Cc: gray....@gsa.gov; philip....@gsa.gov; kin...@gmail.com
Subject: Re: Tackling API Discovery In The Federal Government

 

Kin - you touched on some really solid points about public feedback of government APIs.  That, coupled with Navin's very fruitful request for feedback motivated me to bring together the list of government APIs that have a public GitHub issue tracker set up for them (also pasted below).  As you can see from the feedback being offered up for the SAM API, there's a lot of value for all involved to queue up plenty of feedback for these agencies to work on.  

[Kin Lane]

Happy to publish my feedback to the Github repos. 

Great work Gray!  It would be awesome if every API effort had its own Github issue management and milestones. ;-)

I will work to make habit of publishing my feedback to designated repo (if possible)

You are very welcome Mary. I love doing this, and I'm only critical because I want to see everyone succeed. If I could do all the work myself, I would! ;-)

Thanks!

Kin Lane (API Evangelist)

kin...@gmail.com

@kinlane

--

apievangelist.com

apicommons.org

apivoice.com

--

[David Berlind]

Hey all, sorry for chiming in late here. I was on vacation and just returned. This is a great thread and we at ProgrammableWeb, as many of you know, are very passionate about API discovery and search. So, first, as a reminder, if you have a public API that does not yet surface in any of ProgrammableWeb's search results, please feel free to contact me directly, or visit our page for adding an API. As a side note, we can also make a record of 1st and 3rd party SDKs, Frameworks, and Libraries that stand-alone (eg: Evernote) or that insulate developers from APIs (RESTful or not). Under the hood, our data-model connects these "insulators" to any related APIs in a way that offers ProgrammableWeb's users access to such related resources in a variety of contexts (one of which is API search results and another of which is our editorial content). 

Having run the world's largest API search engine for almost 10 years now, we've learned a thing or two about API search and what users of an API search engine are looking for. 

Today, ProgrammableWeb relies humans to maintain the quality and integrity of its search results. Some APIs come to us by way of the aforementioned Add API page, others come as a result of the research we conduct on a daily basis, and previously, some APIs arrived on our doorstep via ProgrammableWeb's own write-capable API. In any of these cases, we humanly vet all APIs before they are posted to our public directory. You'd be surprised at how many API submissions are nothing of the sort but rather spammers hoping to generate visibility for their non-API-related link bait. Spammers are an ingenious bunch. As evidenced by the millions of dollars that Google spends on its search algorithms (Google has all but ditched metadata as a variable), the spam problem is a cat and mouse game that plagues any 100% machine driven approach. 

This said, no one has felt the pain (and scalability limitations) of human involvement with API search better than ProgrammableWeb which is why we've carefully considered machine-based approaches and why we would like to participate in any conversation around standards (de facto or de jure). After all, ProgrammableWeb's API search engine and the millions of developers who use it every year will get to benefit equally from any open technology or format that helps us to scale API discovery.

Finally, more in direct response to Gray's note, thank you so much Gray for ensuring that ProgrammableWeb gets equal consideration in the government's conversation about API discoverability. In my view, the US government is setting the gold standard for API best practices, standards, and polices that serve as models for all organizations to follow. The speed with which such a large "organization" is racing towards standard, well-informed approaches and the rapidity of API releases across agencies is not only breathtaking, but also worthy of significantly more coverage by ProgrammableWeb's reporters. If any agency or other government constituent out there has API-related announcements, how-tos, advice, etc., please let me know (by email or at the monthly meetups) or write to edi...@programmableweb.com so that we can help get the word out about the fine work that you're doing.

David Berlind

editor-in-chief

ProgrammableWeb.com 

[Gray Brooks]

Thanks, David - 

I've started a section about Ensuring Discoverability on the /Developer Program website.  It's just a start so please share pull requests, as always.  

Gray 

[Gray Brooks]

Mad props to the Trade.gov team, whose continuous updates to their developer portal now includes a http://developer.trade.gov/api.json, where they followed up on one of the issues submitted by a forum member and are now offering machine-readable index of their APIs  

If you haven't yet, def. check out developer.trade.gov and keep the issues and pull requests coming.  

Gray