UK's legislation.gov on opening up british legislation (and the virtues of APIs)
Daniel Schuman
The British are thinking through how to work with the open source / developer community in a way that is truly remarkable. Here the head of legislation services talks about the virtues of building everything on APIs that are made available to the public. -- Daniel
Putting APIs first: legislation.gov.uk
Mention the word “API” to the wrong audience and blank looks shortly follow. For the uninitiated, an API, or“Application Programming Interface”, is a way for one computer to use information or services held on another computer, often across the internet. The strategists say that developing high quality APIs has the potential to transform public services. One example where this is proving to be the case is legislation.gov.uk, the official UK legislation website, operated by The National Archives.
With legislation.gov.uk we aimed to open-up access to the government’s legislation database, by creating an API first. The legislation.gov.uk API allows anyone to access the data we hold in the database, or to use the services we have built, such as the search or to dynamically create PDF documents from the data. We wanted to show that government could create a high-quality, technically sophisticated API, following the principles of both REST and Linked Data. We also wanted our API to be fully open, with no restrictions on use, and no need to register before using it.
We developed the API and then built the legislation.gov.uk website on top of it. The API isn’t a bolt-on or additional feature, it is the beating heart of the service. Thanks to this approach it is very easy to access legislation data – just add /data.xml or /data.rdf to any web page containing legislation, or /data.feed, to any list or search results. One benefit of this approach is that the website, in a way, also documents the API for developers, helping them understand this complex data. Since launching the API we’ve seen several third party applications be developed, including two different iPhone / iPad apps, as well as innovative new products, such as a service for law lecturers to create and self-publish relevant extracts of legislation for their courses.
Many users assume, when they look at a page of legislation on the web, that it is current, in force and applies to where they live. Unfortunately legislation is a lot more complicated than that (this clip from Yes Minister may be an exaggeration, but you’ll get the idea). One Act may amend others, often in complicated ways. A major challenge with operating legislation.gov.uk is keeping pace with all the changes, so we can provide the current view of legislation the public expects. We have an editorial team at The National Archives, who identify all the changes and update the database. We create new versions of the legislation to show how it has changed over time. The difficulty is we can only manage to apply changes to primary legislation (Acts) and the number of changes to apply has been growing every year. It is a mammoth task and we needed to think how we could bring more expertise to bear on this growing challenge.
The API is enabling us to develop an entirely new approach to updating legislation, inspired by the open source software movement – our Expert Participation Programme. We are inviting people from outside The National Archives to work with us, to apply changes to the legislation in the database. Quality is maintained through our editorial practice and a stringent process of review. It is an exciting time as private sector companies are now investing, employing people to work with us to bring the database up to date and to maintain it. The API gives them ready and easy access to the data, which they can include in their own products and services. To help people use the data we have also published the code we use to present the content in HTML and PDF on github: https://github.com/legislation/legislation. Importantly, everything remains public, open and free, under the Open Government Licence. It is a win-win situation.
The legislation.gov.uk API has changed everything for us. It powers our website. It has enabled us to move to an open data business model, securing the editorial effort we need from the private sector for this important source of public data. It allows us to deliver information and services across channels and platforms through third party applications. We are developing other tools that use the API, using Linked Data – from recording the provenance of new legislation as it is converted from one format to another, to a suite of web based editorial tools for legislation, including a natural language processing capability that automatically identifies the legislative effects. Everything we do is underpinned by the API and Linked Data. With the foundations in place, the possibilities of what can be done with legislation data are now almost limitless.
About the author: John Sheridan is Head of Legislation Services at The National Archives
Daniel
Daniel Schuman
Director | Advisory Committee on Transparency
Policy Counsel | The Sunlight Foundation
o: 202-742-1520 x 273 | c: 202-713-5795 | @danielschuman
Eric Mill
--
You received this message because you are subscribed to the Google Groups "Open House Project" group.
To post to this group, send email to openhous...@googlegroups.com.
To unsubscribe from this group, send email to openhouseproje...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/openhouseproject?hl=en.
Eric Mill
Daniel Schuman
Daniel
Daniel Schuman
Director | Advisory Committee on Transparency
Policy Counsel | The Sunlight Foundation
o: 202-742-1520 x 273 | c: 202-713-5795 | @danielschuman
Gregory Slater
Daniel Schuman
Daniel
Daniel Schuman
Director | Advisory Committee on Transparency
Policy Counsel | The Sunlight Foundation
o: 202-742-1520 x 273 | c: 202-713-5795 | @danielschuman
Gregory Slater
Josh Tauberer
> 1. It seems that there is little comprehension or at least interest
> within the responsible people on the US side, of the crucial importance
> of markup language for making the data truly machine searchable
It depends on where you look. Lumping the whole government together
loses the important milestones. Within the legislative branch, the House
Majority Leader's office and the staff of the Committee on House
Administration get it 100%. As Daniel mentioned, they launched
docs.house.gov this year and I believe are actively working on it still.
Does the Senate get it? Probably not. And the Library of Congress?
Definitely not.
In the executive branch we're in the middle of a lull right now because
of new people coming in as the federal CTO and CIO. The new CTO was just
named a few weeks ago-- it's Todd Park, who was the CTO of the Dept. of
Health and Human Services. There wasn't a better person for the job.
Under Park, HHS became the one agency that I know of to sincerely
embrace data since the 2009 open gov directive. Last year three or four
people from HHS attended Transparency Camp. This year there will be the
*third* Health Datapalooza (hdiforum.org). (Disclaimer: I'm currently
consulting for HHS, but I've been saying this for a long time.)
To talk about the US as a whole, we should look at the states too. Great
things are coming out of Chicago, Boston, and New York City lately,
especially because they've hired the right people to lead those efforts.
Alex Howard has covered this:
http://radar.oreilly.com/2011/08/chicago-data-apps-open-government.html
http://radar.oreilly.com/2011/10/data-new-york-city.html
So if you look in the right places, you can actually find a lot of
exciting things.
I'm relatively happy with the progress open gov data has been making in
the last couple of years, when you look at the nation as a whole. (It's
important to remember that, legally speaking, the House, Senate, exec
branch, and the states are all operating independently. And within each
responsibilities are spread around, making it hard to point to a single
place to look for whether people get it.)
It's just a little sad given this progress that the most basic
information, the primary legal documents, still face such a hurdle for
being made completely open. I'm throwing in here the status of
legislation since that's my hobby horse, but judicial documents remain a
mess and have made no progress since the dawn of the Internet, and
state-level legal documents are hit and miss (see law.resource.org).
(At this point my computer crashed, so I had better send the saved draft
before anything explodes.)
- Josh Tauberer (@JoshData)
On 04/02/2012 04:08 AM, Gregory Slater wrote:
>
> Hi Daniel,
>
> Thanks very much for this quick take. I appreciate it, as well as
> anything anyone else has to add.
> A couple comments:
>
> 1. It seems that there is little comprehension or at least interest
> within the responsible people on the US side, of the crucial importance
> of markup language for making the data truly machine searchable
> 2. Perhaps establishing an N-parameter ranking of transparency progress
> by various nations would provide some motivation. Is there a
> 'Transparency Watch' analogous to 'Human Rights Watch'?
> 3. What about setting up an 'X Prize's for transparency goals? (no idea
> where to get funding…)
>
> Anyway, thanks much,
>
> - Greg Slater
>
>
>
>
>
> On Apr 1, 2012, at 2:15 PM, Daniel Schuman wrote:
>
>> Greg, I'll give you a very quick overview, although to large extent,
>> it depends on what realm we're talking about. (I also reverse the
>> right to extent/ modify these comments as I think about this some
>> more. I hope that Josh and Eric and others will chime in, too.)
>>
>> From a legislative perspective, the Congress basically has not entered
>> the API/ Bulk data game, whereas the UK is making available APIs
>> <http://data.gov.uk/dataset/legislation-api>. We've seen an initiative
>> in the past year from the House of Representatives
>> <http://sunlightfoundation.com/blog/2012/01/13/house-launches-transparency-portal/>
>> which is moving quickly towards bulk access to leg data, but that's
>> only for the House -- we still need the Senate, and most importantly,
>> the legislative support agencies (i.e. the Library of Congress).
>> Nearly all bills are now available from the Congress as XML (via GPO
>> on THOMAS), but you can't get API/BULK bill status info, cosponsorship
>> info, etc., etc. (GPO and the Clerk are responsible for the XML.) We
>> are all still relying on Josh for his scrapings of THOMAS for the
>> contextual information. I had a great chat with the British leadership
>> when they were here a few weeks ago, and they seem to have a genuine
>> commitment to discussing how they can bring in civil society
>> organizations, making their data available better, etc. The LOC's
>> focus for now appears to be on a revamp of all of their websites, not
>> providing info in more developer-friendly formats. (GPO seems to be
>> more in favor of improving access, and maintains several datasets in
>> bulk <http://www.gpo.gov/fdsys/bulkdata>, although it's really the
>> exec branch side. I also don't have a good way to evaluate FDSys,
>> although I've heard from programmers that they like the back end.)
>>
>> On the executive branch side, there's some progress on both sides. The
>> US was the first to launch data.gov <http://data.gov/>, and it has
>> around 400,000 datasets, of which maybe 1-2k are things that a
>> significant number of people might use. The biggest problem on the US
>> side is that many agencies are incredibly slow to put info on data.gov
>> <http://data.gov/>, particularly the stuff that people really want to
>> use, if it is at all controversial. I'm not in a position to evaluate
>> the British effort for data.gov.uk <http://data.gov.uk/> -- you'd
>> probably have to ask someone from MySociety. There's a huge ongoing
>> effort to evaluate data.gov <http://data.gov/> and the open government
>> directive generally, and I'd hate to oversimplify too much.
>>
>> As for the judiciary, I can't speak to the British experience, but the
>> US Courts generally haven't bought into the idea of free public access
>> to leg material, leg alone bulk or api access. Others on the list
>> could speak to this better than I.
>>
>> I'm sure I've missed important things, elided over an area of
>> significant progress, etc., but here's a quick first stab.
>>
>>
>> Daniel
>>
>> Daniel Schuman
>> Director | Advisory Committee on Transparency
>> <http://transparencycaucus.org/>
>> Policy Counsel | The Sunlight Foundation <http://sunlightfoundation.com/>
>> o: 202-742-1520 x 273 | c: 202-713-5795 | @danielschuman
>>
>>
>> On Sun, Apr 1, 2012 at 2:34 PM, Gregory Slater <ten...@pacbell.net
>> <mailto:ten...@pacbell.net>> wrote:
>>
>>
>>
>> Hello Daniel Schuman, Josh, and other experts,
>>
>> Could you succinctly compare US and UK governments' current
>> transparency status when it comes to APIs, machine readability,
>> and data online (or point me to such a comparison)?
>>
>> Thanks,
>>
>> - Greg Slater
>>
>>
>>
>> On Apr 1, 2012, at 9:21 AM, Daniel Schuman wrote:
>>
>>> They can't keep up with integrating passed legislation into their
>>> code, so they're asking the private sector to help them do that.
>>>
>>> Daniel
>>>
>>> Daniel Schuman
>>> Director | Advisory Committee on Transparency
>>> <http://transparencycaucus.org/>
>>> Policy Counsel | The Sunlight Foundation
>>> <http://sunlightfoundation.com/>
>>> o: 202-742-1520 x 273 <tel:202-742-1520%20x%20273> | c:
>>> 202-713-5795 <tel:202-713-5795> | @danielschuman
>>>
>>>
>>> On Fri, Mar 30, 2012 at 6:43 PM, Eric Mill
>>> <er...@sunlightfoundation.com
>>> <mailto:er...@sunlightfoundation.com>> wrote:
>>>
>>> "It has enabled us to move to an open data business model,
>>> securing the editorial effort we need from the private sector
>>> for this important source of public data."
>>>
>>> Sorry, Gmail's highlighting cut off my excerpt. The above
>>> line is what I was asking about.
>>>
>>>
>>> On Fri, Mar 30, 2012 at 6:42 PM, Eric Mill
>>> <er...@sunlightfoundation.com
>>> <mailto:er...@sunlightfoundation.com>> wrote:
>>>
>>> What does this mean, do you think?
>>>
>>> "It has enabled us to move to an open data business
>>> model, securing the editorial effort we need from the
>>> private sector for this important source of public data."
>>>
>>> On Fri, Mar 30, 2012 at 5:01 PM, Daniel Schuman
>>> <dsch...@sunlightfoundation.com
>>> <mailto:dsch...@sunlightfoundation.com>> wrote:
>>>
>>>
>>> The British are thinking through how to work with
>>> the open source / developer community in a way that
>>> is truly remarkable. Here the head of legislation
>>> services talks about the virtues of building
>>> everything on APIs that are made available to the
>>> public. -- Daniel
>>>
>>>
>>>
>>> Putting APIs first: legislation.gov.uk
>>> <http://legislation.gov.uk/>
>>>
>>>
>>> by johnlsheridan on 30/03/2012
>>> /http://digital.cabinetoffice.gov.uk/2012/03/30/putting-apis-first-legislation-gov-uk/
>>> /
>>> legislationgovuk
>>>
>>> Mention the word “API” to the wrong audience and
>>> blank looks shortly follow. For the uninitiated, an
>>> API, or“Application Programming Interface”
>>> <http://en.wikipedia.org/wiki/Api>, is a way for one
>>> computer to use information or services held on
>>> another computer, often across the internet. The
>>> strategists say that developing high quality APIs has
>>> the potential to transform public services. One
>>> example where this is proving to be the case is
>>> legislation.gov.uk <http://www.legislation.gov.uk/>,
>>> the official UK legislation website, operated by The
>>> National Archives.
>>>
>>>
>>> With legislation.gov.uk
>>> <http://www.legislation.gov.uk/> we aimed to open-up
>>> access to the government’s legislation database, by
>>> creating an API first. The legislation.gov.uk API
>>> <http://www.legislation.gov.uk/developer/contents>
>>> allows anyone to access the data we hold in the
>>> database, or to use the services we have built, such
>>> as the search or to dynamically create PDF documents
>>> from the data. We wanted to show that government
>>> could create a high-quality, technically
>>> sophisticated API, following the principles of both
>>> REST
>>> <http://www.ics.uci.edu/~fielding/pubs/dissertation/rest_arch_style.htm>
>>> and Linked Data <http://linkeddata.org/>. We also
>>> wanted our API to be fully open, with no restrictions
>>> on use, and no need to register before using it.
>>>
>>> We developed the API and then built the
>>> legislation.gov.uk <http://legislation.gov.uk/>
>>> website on top of it. The API isn’t a bolt-on or
>>> additional feature, it is the beating heart of the
>>> service. Thanks to this approach it is very easy to
>>> access legislation data – just add /data.xml or
>>> /data.rdf to any web page containing legislation, or
>>> /data.feed, to any list or search results. One
>>> benefit of this approach is that the website, in a
>>> way, also documents the API for developers, helping
>>> them understand this complex data. Since launching
>>> the API we’ve seen several third party applications
>>> be developed, including two different iPhone / iPad
>>> apps, as well as innovative new products, such as a
>>> service for law lecturers to create and self-publish
>>> relevant extracts of legislation for their courses.
>>>
>>> Many users assume, when they look at a page of
>>> legislation on the web, that it is current, in force
>>> and applies to where they live. Unfortunately
>>> legislation is a lot more complicated than that (this
>>> clip from Yes Minister
>>> <http://www.youtube.com/watch?v=Eikb2lX5xYE> may be
>>> an exaggeration, but you’ll get the idea). One Act
>>> may amend others, often in complicated ways. A major
>>> challenge with operating legislation.gov.uk
>>> <http://legislation.gov.uk/> is keeping pace with all
>>> The legislation.gov.uk <http://legislation.gov.uk/>
>>> API has changed everything for us. It powers our
>>> website. It has enabled us to move to an open data
>>> business model, securing the editorial effort we need
>>> from the private sector for this important source of
>>> public data. It allows us to deliver information and
>>> services across channels and platforms through third
>>> party applications. We are developing other tools
>>> that use the API, using Linked Data – from recording
>>> the provenance of new legislation as it is converted
>>> from one format to another, to a suite of web based
>>> editorial tools for legislation, including a natural
>>> language processing capability that automatically
>>> identifies the legislative effects. Everything we do
>>> is underpinned by the API and Linked Data. With the
>>> foundations in place, the possibilities of what can
>>> be done with legislation data are now almost limitless.
>>>
>>> /About the author: John Sheridan is Head of
>>> Legislation Services at The National Archives/
>>>
>>>
>>> Daniel
>>>
>>> Daniel Schuman
>>> Director | Advisory Committee on Transparency
>>> <http://transparencycaucus.org/>
>>> Policy Counsel | The Sunlight Foundation
>>> <http://sunlightfoundation.com/>
>>> o: 202-742-1520 x 273 <tel:202-742-1520%20x%20273> |
>>> c: 202-713-5795 <tel:202-713-5795> | @danielschuman
>>>
>>> --
>>> You received this message because you are subscribed
>>> to the Google Groups "Open House Project" group.
>>> To post to this group, send email to
>>> openhous...@googlegroups.com
>>> <mailto:openhous...@googlegroups.com>.
>>> To unsubscribe from this group, send email to
>>> openhouseproje...@googlegroups.com
>>> <mailto:openhouseproject%2Bunsu...@googlegroups.com>.
>>> For more options, visit this group at
>>> http://groups.google.com/group/openhouseproject?hl=en.
>>>
>>>
>>>
>>>
>>> --
>>> Developer | sunlightfoundation.com
>>> <http://sunlightfoundation.com/>
>>>
>>>
>>>
>>>
>>> --
>>> Developer | sunlightfoundation.com
>>> <http://sunlightfoundation.com/>
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the
>>> Google Groups "Open House Project" group.
>>> To post to this group, send email to
>>> openhous...@googlegroups.com
>>> <mailto:openhous...@googlegroups.com>.
>>> To unsubscribe from this group, send email to
>>> openhouseproje...@googlegroups.com
>>> <mailto:openhouseproject%2Bunsu...@googlegroups.com>.
>>> For more options, visit this group at
>>> http://groups.google.com/group/openhouseproject?hl=en.
>>>
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the
>>> Google Groups "Open House Project" group.
>>> To post to this group, send email to
>>> openhous...@googlegroups.com
>>> <mailto:openhous...@googlegroups.com>.
>>> To unsubscribe from this group, send email to
>>> openhouseproje...@googlegroups.com
>>> <mailto:openhouseproje...@googlegroups.com>.
>>> For more options, visit this group at
>>> http://groups.google.com/group/openhouseproject?hl=en.
>>
>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Open House Project" group.
>> To post to this group, send email to
>> openhous...@googlegroups.com
>> <mailto:openhous...@googlegroups.com>.
>> To unsubscribe from this group, send email to
>> openhouseproje...@googlegroups.com
>> <mailto:openhouseproject%2Bunsu...@googlegroups.com>.
>> For more options, visit this group at
>> http://groups.google.com/group/openhouseproject?hl=en.
>>
>>
>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Open House Project" group.
>> To post to this group, send email to openhous...@googlegroups.com
>> <mailto:openhous...@googlegroups.com>.
>> To unsubscribe from this group, send email to
>> openhouseproje...@googlegroups.com
>> <mailto:openhouseproje...@googlegroups.com>.
Daniel Schuman
Daniel
Daniel Schuman
Director | Advisory Committee on Transparency
Policy Counsel | The Sunlight Foundation
o: 202-742-1520 x 273 | c: 202-713-5795 | @danielschuman
Eric Mill
Hi Daniel,Thanks very much for this quick take. I appreciate it, as well as anything anyone else has to add.A couple comments:1. It seems that there is little comprehension or at least interest within the responsible people on the US side, of the crucial importance of markup language for making the data truly machine searchable
2. Perhaps establishing an N-parameter ranking of transparency progress by various nations would provide some motivation. Is there a 'Transparency Watch' analogous to 'Human Rights Watch'?
Gregory Slater
Hello Josh. Eric, and Daniel,
Thanks very much for the comments, links, and perspective. Obviously my sweeping statement was pretty much ridiculous and ignorance-based. You all and many others have worked long and hard to increase gov transparency, and I thank you for your efforts.
I follow the openhouseproject thread and so was more or less aware of the successes on the House side apparently due to Boehner's backing (god it should humiliating to the dem leadership that the repubs led on this).
That said, it is not surprising to me that the House is pushing transparency and the Senate is resisting. The closer you get to real power the greater the resistance to genuine transparency. Information is power - that's why the powerful - senators, presidents, justices, etc, resist most strongly. That's why the "Constitution Annotated" is redacted and vandalized before release at great expense and trouble. That's why Obama is persecuting Bradley Manning. As I have said before, the real information we need to oversee our employees (President, Senators, SCOTUS) will be the information that is most jealously guarded. They're all Machiavellians - throw-backs to the sixteenth century. The question is - will they be able to control our printing press (the Internet and universal connectivity and access) in a way the Catholic Church was unable to do in Martin Luther's age? The printing press ushered in the Reformation and then the Enlightenment, but Barack Obama, the SCOTUS, and the Congress, as well as their corporate overlords, are intent on learning from the strategic blunders made by Pope Leo X five hundred years ago.
Thanks for all you do,
- greg slater