New Memorandum almost defines open data, misses mark with open licenses
Josh Tauberer
http://razor.occams.info/blog/2013/05/09/new-open-data-memorandum-almost-defines-open-data-misses-mark-with-open-licenses/
New Open Data Memorandum almost defines open data, misses mark with open licenses
TL:DR: The new E.O. and memorandum are good for transparency and lock in almost all of the generally accepted notions of open government data. But it misses the mark on the requirement of “open licenses.”
With an executive order and a new Memorandum on Open Data Policy today, the focus on entrepreneurship remained at the forefront of federal data policy. This focus began with last year’s Digital Government Strategy, and these days weather data and GPS signals are the examples of choice. That said, the policies set in the new memorandum are quite good for the classic use of this data (transparency, accountability, and civic education) even if “transparency” is only barely mentioned in passing.
This new Open Data Memorandum presents the most detailed definition to date of “open data” by the federal government. It included many of the principles that our community has reached consensus on, but it gets one severely wrong.
As I wrote many years ago, the 2009 Open Government Directive itself already adopted some of the principles of open government data including: online, primary, timely, public input, and public review. It also added two principles of its own: being pro-active about data release and creating accountability by designating an official responsible for data quality.
Comparing to my list of open government data principles in my book, the new memorandum’s definition of open data covers:
- Principle 1: Information should be online (to quote the Memorandum: “retrieved, downloaded”)
- Principle 2: Primary (the Memorandum even uses language from the 8 Principles; interestingly the memorandum places this under the heading of “Complete,” which was a different principle from the original 8 Principles).
- Principle 3: Timely.
- Principle 4: Accessible (the Memorandum repeats the language from the 8 Principles, “available to the widest range of users for the widest range of purposes” and the use of “multiple formats” where necessary, and for documentation says the data should be “described”).
- Principles 5 and 10: Analyzable (“machine readable”).
- Principle 6: Non-discriminatory
- Principle 7: Non-proprietary (open) data formats
- Principle 14: Public review (“A point of contact must be designated to assist with data use and to respond to complaints about adherence to these open data requirements.”)
Its definition also states that open data has a presumption of openness. (Principles 2-7 and 14 are from the 8 Principles of Open Government Data. Principle 1 is from the Sunlight Foundation.)
Elsewhere in the memorandum it addresses:
- Principle 13: Public input (“engage with customers” for prioritizing what data should be made available and how to make it available)
- Principle 15. Interagency coordination (“interoperability”)
It also asks agencies to create data catalogs to include datasets “that can be made publicly available but have not yet been released” at agency.gov/data URLs. And it says agencies must consider the needs of open data at all stages of the information collection lifecycle. In other words, data should be collected in such a way as to promote public dissemination of open data later on.
The Memorandum misses the principle that data should be license-free, which is a core principle and a grave mistake. It also misses the peripheral principles of permanence, the use of safe file formats, and practices of provenance and trust (e.g. digital signatures). (These last two are ACM principles.)
Rather than requiring open data to be license-free, which was a core part of the 8 Principles of Open Government Data, it instead promotes the use of “open licenses.” This is a subtle but important distinction. Licenses presume data rights. Open licenses, including open source licenses and Creative Commons licenses, create limited privileges in a world where the default is closed. These licenses create possibilities of use that do not exist in the absence of the license because copyright law, or other law, creates an initial state of closedness.
Most open licenses only grant some privileges but not others, and some privileges come along with new requirements. The GPL and Creative Commons Attribution License, for instance, rely on copyright law so that restrictions on data use intended by the open license (GPL’s virality clause, or the restriction that users must attribute the work to the author) are enforceable in court.
Federal government data is not typically subject to copyright law, and in this case a license is not needed for the data to be open. Thus the application of a license suggests a change from the open-by-default state of this data to a closed-by-default state where a license is required to open it up. While the memorandum requires “an open license that places no restrictions on their [dataset] use,” the term “open license” is typically understood to presume a default closed state. This policy opens the door (so to speak) to agencies applying licenses (i.e. new contractual agreements) to data that serve only to restrict use.
Federal government data not subject to copyright cannot be free if a license is applied. The license-free principle of the original 8 Principles says open government data cannot be limited in this way.
When data may be subject to copyright protection (copyright law is murky and there are many gray areas), or when copyright law definitely applies (such as to documents produced originally by federal government contractors), then a public domain dedication such as the Creative Commons CC0 statement is appropriate. A public domain dedication differs from an open license in that it disclaims copyright and other protections, whereas, again, an open license implies that such a limitation on use is already present. The CC0 statement was successfully used by the Council of the District of Columbia to disclaim copyright over data files containing the DC Code.
Finally, while the definition of open data is otherwise quite strong, the definition is used just once in the whole memorandum. The memorandum does not mandate that government data be open data under its definition, at least as far as I could see. The only use of the open data definition is in its request for agencies to create roles for staff to ensure data released to the public are open. That is, staff should promote open data, but open data itself is not required.
While the goals of the Memorandum in defining open data and using open licenses are laudable, the implementation does not meet the 8 Principles requirements of open government data, at least under the usual understanding of “open license,” and the use of the definition to promote open data is very limited.
PS. As Derek Willis points out over Twitter, the “mosaic effect” paragraphs in the memorandum are also somewhat concerning. The mosaic effect is hard to quantify and therefore difficult to limit, and this creates a big hole for keeping data government out of public reach.
David Robinson
--
You received this message because you are subscribed to the Google Groups "Open House Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openhouseproje...@googlegroups.com.
To post to this group, send email to openhous...@googlegroups.com.
Visit this group at http://groups.google.com/group/openhouseproject?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
John Wonderlich
Gunnar Hellekson
For the data that's in the public domain, my understanding is that the Federal government couldn't license it even if they wanted to.
Other data, probably created through contractors, may indeed carry a copyright -- I'm thinking of traffic.com as the worst case. For that data, insisting on an open license makes sense.
g
Gavin Baker
But under some interpretations, what I said above only applies within the U.S. In other words, the federal government could technically claim copyright on its works when used abroad. So there is, in theory, something to license here.
Personally, I think that would be bad policy: to create confusion and uncertainty for U.S. users in order to potentially restrict users abroad. But it would be legal, according to some interpretations.
With that said, I think the intent of the clause here is to say, "Make clear that there are no restrictions on using this information." I agree with that goal, and in fact we recommended it in our 21st Century Right-to-Know recommendations from 2008. The OMB memo just says it in a slightly sub-optimal way.
Gavin Baker
Policy Analyst, Open Government Policy
Center for Effective Government (formerly OMB Watch)
gba...@foreffectivegov.org
Phone: (202) 683-4834
Twitter: @opengavin
LinkedIn: gavinrbaker
-----Original Message-----
From: Gunnar Hellekson [mailto:gunnar.h...@gmail.com]
Sent: Thursday, May 09, 2013 1:47 PM
To: openhous...@googlegroups.com
Subject: Re: [openhouseproject] New Memorandum almost defines open data, misses mark with open licenses
For the data that's in the public domain, my understanding is that the Federal government couldn't license it even if they wanted to.
Other data, probably created through contractors, may indeed carry a copyright -- I'm thinking of traffic.com as the worst case. For that data, insisting on an open license makes sense.
g
On 9 May 2013, at 12:43, John Wonderlich wrote:
> There is a funny tension between encouraging the use of "open licenses" (as though the default for US government data were anything else but public domain) while also saying "and no restrictions should be placed upon their use" throughout the document (which is the appropriate standard).
>
>
> On Thu, May 9, 2013 at 1:24 PM, Josh Tauberer <taub...@govtrack.us> wrote:
> My usual evaluation on principles...
>
> http://razor.occams.info/blog/2013/05/09/new-open-data-memorandum-almo
> st-defines-open-data-misses-mark-with-open-licenses/
>
> New Open Data Memorandum almost defines open data, misses mark with
> open licenses
> TL:DR: The new E.O. and memorandum are good for transparency and lock in almost all of the generally accepted notions of open government data. But it misses the mark on the requirement of "open licenses."
>
> With an executive order and a new Memorandum on Open Data Policy today, the focus on entrepreneurship remained at the forefront of federal data policy. This focus began with last year's Digital Government Strategy, and these days weather data and GPS signals are the examples of choice. That said, the policies set in the new memorandum are quite good for the classic use of this data (transparency, accountability, and civic education) even if "transparency" is only barely mentioned in passing.
>
> This new Open Data Memorandum presents the most detailed definition to date of "open data" by the federal government. It included many of the principles that our community has reached consensus on, but it gets one severely wrong.
>
> As I wrote many years ago, the 2009 Open Government Directive itself already adopted some of the principles of open government data including: online, primary, timely, public input, and public review. It also added two principles of its own: being pro-active about data release and creating accountability by designating an official responsible for data quality.
>
> Comparing to my list of open government data principles in my book, the new memorandum's definition of open data covers:
>
> o Principle 2: Primary (the Memorandum even uses language from the 8 Principles; interestingly the memorandum places this under the heading of "Complete," which was a different principle from the original 8 Principles).
> o Principle 3: Timely.
> o Principle 4: Accessible (the Memorandum repeats the language from the 8 Principles, "available to the widest range of users for the widest range of purposes" and the use of "multiple formats" where necessary, and for documentation says the data should be "described").
> o Principles 5 and 10: Analyzable ("machine readable").
> o Principle 6: Non-discriminatory
> o Principle 7: Non-proprietary (open) data formats
> o Principle 14: Public review ("A point of contact must be designated
> to these open data requirements.") Its definition also states that
> open data has a presumption of openness. (Principles 2-7 and 14 are
> from the 8 Principles of Open Government Data. Principle 1 is from the
> Sunlight Foundation.)
>
> Elsewhere in the memorandum it addresses:
>
> o Principle 15. Interagency coordination ("interoperability")
Eric Mill
JAMES JACOBS
17 USC § 105 - Subject matter of copyright: United States Government worksCopyright protection under this title is not available for any work of the United States Government, but the United States Government is not precluded from receiving and holding copyrights transferred to it by assignment, bequest, or otherwise.
CopyrightPublic Domain Notices: A Good Idea?When we depend on pointing instead of collectingUS government sends itself a takedown noticeWait! Don't digitize and discard!Complications of the U.S. Public DomainWhite House claims copyright of photos on Flickr
WashingtonWatch.com
President Obama’s New E.O.: Open Data, Not Government Transparency
There’s a powerful irony lurking underneath the executive order and OMB memorandum on open data that the White House released in tandem today: We don’t have data that tells us what agencies will carry out these policies.
It’s nice that the federal government will work more assiduously to make available the data it collects and creates. And what President Obama’s executive order says is true: “making information resources easy to find, accessible, and usable can fuel entrepreneurship, innovation, and scientific discovery that improves Americans’ lives and contributes significantly to job creation.” GPS and weather data are the premier examples.
But government transparency was the crux of the president’s 2008 campaign promises, and it is still the rightful expectation of the public. Government transparency is not produced by making interesting data sets available. It’s produced by publishing data about the government’s deliberations, management, and results.
Today’s releases make few, if any, nods to that priority. They don’t go to the heart of transparency, but threaten to draw attention away from the fact that basic data about our government, including things as fundamental as the organization of the executive branch of government, are not available as open data.
Yes, there is still no machine-readable government organization chart. This was one of the glaring faults we found when we graded the publication practices of Congress and the executive branch last year, and this fault remains. The coders who may sift through data published by various agencies, bureaus, programs, and projects can’t sift through data reflecting what those organizational units of government are.
Compare today’s policy announcements to events coming up on Capitol Hill in the next two weeks.
On Thursday next week (May 16), the House Committee on Oversight and Government Reform will host a “DATA Demonstration Day” to illustrate to Congress and the media how technology may cut waste and improve oversight if federal spending data is structured and transparent. (That would include my hobby-horse, the machine-readable federal government organization chart.) We’ll be there demo-ing how we add data to the bills Congress publishes.
On May 22nd, the House Administration Committee is hosting its 2013 Legislative Data and Transparency Conference. This is an event at which various service providers to the House will announce not just policies, but recent, new, and upcoming improvements in publication of data about the House and its deliberations. (We’ll be there, too.)
The administration’s open data announcements are entirely welcome. Some good may come from these policies, and they certainly do no harm (barring procurement boondoggles–which, alas, is a major caveat). But I hope this won’t distract from the effort to produce government transparency, which I view as quite different from the subject of the new executive order and memorandum. The House of Representatives still seems to be moving forward on government transparency with more alacrity.
Josh Tauberer
On 05/09/2013 01:42 PM, David Robinson wrote:
This restriction does not stop (or at least, does not obviously stop) the federal government from imposing "terms of use" on .gov web sites, and there remains some risk that such terms could attempt to impose restrictions tantamount to copyright.
On 05/09/2013 01:47 PM, Gunnar Hellekson wrote:
Other data, probably created through contractors, may indeed carry a copyright -- I'm thinking of traffic.com as the worst case. For that data, insisting on an open license makes sense.
For the moment I'm standing by that.
On 05/09/2013 02:50 PM, JAMES JACOBS wrote:
a "government work" may incorporate information that is *not* a work of the government and the agency responsible for the government publication may leave it up to the user to figure out what is open access and what is not.
I've created two issues on project-open-data, where apparently OSTP wants feedback (let's see if they address any of it):
https://github.com/project-open-data/project-open-data.github.io/issues/5
https://github.com/project-open-data/project-open-data.github.io/issues/8
Josh
Josh Tauberer
>
> Today’s releases make few, if any, nods to that priority. They don’t
> go to the heart of transparency, but threaten to draw attention away
> from the fact that basic data about our government, including things
> as fundamental as the organization of the executive branch of
> government, are not available as open data.
>
be judged by (i.e. it's not a transparency policy), but I'm totally with
you here. The EO and memorandum today continue the trend since (gasp)
Todd Park's appointment, if not earlier, that completely sidelines
transparency. And it fits entirely in line with other trends of this
administration which John has blogged about a lot.
Which isn't to say I don't think Todd Parkian (I forget who coined that)
entrepreneurial innovation is important. Double negative. I think it's
important. I'm glad we have this angle on open data. I like seeing
#opengov grow into a wider concept.
And the new policy is certainly helpful for transparency (aside from
Derek's concerns on Twitter).
But the culture change that began in 2009 isn't going to continue
without the rah-rah transparency rhetoric.
- Josh Tauberer (@JoshData)
http://razor.occams.info
Greg Elin
It would be great to have some get togethers to discuss the policy and
thoughts. Maybe a Sunday Brunch?
It is not a secret this administration has appointed a variety of
leaders who believe default settings matter. The new policy formally
sets the default to "open" using a definition of open originating from
this community.
The policy and its presentation is a major milestone. It was reached
by the remarkable civic engagement of this community, endless work by
this administration and decades of effort by others inside and outside
towards e-government.
I personally believe we are on an extended journey modernizing
institutions built on the printing press to institutions compatible
with the networked computer. If you've ever have driven cross-country
from New York to California, arriving in Denver is an interesting
moment. You realize you have come incredibly far. Then you realize you
are still just 2/3 of the way across the continent.
Greg Elin
Sent from my iPhone
Josh Tauberer
> The new policy formally
> sets the default to "open" using a definition of open originating from
> this community.
Peggy Garvin
My question: If a change to the policy is accepted on GitHub, will an amended OMB memorandum attachment be re-released formally? Or will revisions be made solely on Project Open Data? What’s the connection between Project Open Data and the OMB memorandum?
Peggy
On 5/9/13 2:28 PM, "Eric Mill" <er...@sunlightfoundation.com> wrote:
Also, I know CC0 gets a lot of linkage, but something I discovered more recently is CC's "Public Domain Mark":
http://creativecommons.org/choose/mark/
Where CC0 is for releasing data into the public domain (implicitly indicating you could choose another license if you want), the Public Domain Mark indicates data is already in the public domain.
So if the government wants to start dropping license text next to all its datasets to indicate its public domain status, and they want it to look all hip and be associated with Creative Commons, they should use the Public Domain Mark.
On Thu, May 9, 2013 at 2:08 PM, Gavin Baker <gba...@foreffectivegov.org> wrote:
Federal government works are in the public domain (no copyright). If there's no copyright, it can't be licensed. (For those familiar with Creative Commons licenses, the most appropriate option here, CC Zero, is technically a "waiver," not a "license.")
But under some interpretations, what I said above only applies within the U.S. In other words, the federal government could technically claim copyright on its works when used abroad. So there is, in theory, something to license here.
Personally, I think that would be bad policy: to create confusion and uncertainty for U.S. users in order to potentially restrict users abroad. But it would be legal, according to some interpretations.
With that said, I think the intent of the clause here is to say, "Make clear that there are no restrictions on using this information." I agree with that goal, and in fact we recommended it in our 21st Century Right-to-Know recommendations from 2008. The OMB memo just says it in a slightly sub-optimal way.
Gavin Baker
Policy Analyst, Open Government Policy
Center for Effective Government (formerly OMB Watch)
gba...@foreffectivegov.org
Phone: (202) 683-4834 <tel:%28202%29%20683-4834>
Twitter: @opengavin
LinkedIn: gavinrbaker
-----Original Message-----
From: Gunnar Hellekson [mailto:gunnar.h...@gmail.com]
Sent: Thursday, May 09, 2013 1:47 PM
To: openhous...@googlegroups.com
Subject: Re: [openhouseproject] New Memorandum almost defines open data, misses mark with open licenses
For the data that's in the public domain, my understanding is that the Federal government couldn't license it even if they wanted to.
Other data, probably created through contractors, may indeed carry a copyright -- I'm thinking of traffic.com <http://traffic.com> as the worst case. For that data, insisting on an open license makes sense.
> It also asks agencies to create data catalogs to include datasets "that can be made publicly available but have not yet been released" at agency.gov/data <http://agency.gov/data> URLs. And it says agencies must consider the needs of open data at all stages of the information collection lifecycle. In other words, data should be collected in such a way as to promote public dissemination of open data later on.
>
> The Memorandum misses the principle that data should be license-free,
> which is a core principle and a grave mistake. It also misses the
> peripheral principles of permanence, the use of safe file formats, and
> practices of provenance and trust (e.g. digital signatures). (These
> last two are ACM principles.)
>
> Rather than requiring open data to be license-free, which was a core part of the 8 Principles of Open Government Data, it instead promotes the use of "open licenses." This is a subtle but important distinction. Licenses presume data rights. Open licenses, including open source licenses and Creative Commons licenses, create limited privileges in a world where the default is closed. These licenses create possibilities of use that do not exist in the absence of the license because copyright law, or other law, creates an initial state of closedness.
>
> Most open licenses only grant some privileges but not others, and some privileges come along with new requirements. The GPL and Creative Commons Attribution License, for instance, rely on copyright law so that restrictions on data use intended by the open license (GPL's virality clause, or the restriction that users must attribute the work to the author) are enforceable in court.
>
> Federal government data is not typically subject to copyright law, and in this case a license is not needed for the data to be open. Thus the application of a license suggests a change from the open-by-default state of this data to a closed-by-default state where a license is required to open it up. While the memorandum requires "an open license that places no restrictions on their [dataset] use," the term "open license" is typically understood to presume a default closed state. This policy opens the door (so to speak) to agencies applying licenses (i.e. new contractual agreements) to data that serve only to restrict use.
>
> Federal government data not subject to copyright cannot be free if a license is applied. The license-free principle of the original 8 Principles says open government data cannot be limited in this way.
>
> When data may be subject to copyright protection (copyright law is murky and there are many gray areas), or when copyright law definitely applies (such as to documents produced originally by federal government contractors), then a public domain dedication such as the Creative Commons CC0 statement is appropriate. A public domain dedication differs from an open license in that it disclaims copyright and other protections, whereas, again, an open license implies that such a limitation on use is already present. The CC0 statement was successfully used by the Council of the District of Columbia to disclaim copyright over data files containing the DC Code.
>
> Finally, while the definition of open data is otherwise quite strong, the definition is used just once in the whole memorandum. The memorandum does not mandate that government data be open data under its definition, at least as far as I could see. The only use of the open data definition is in its request for agencies to create roles for staff to ensure data released to the public are open. That is, staff should promote open data, but open data itself is not required.
>
> While the goals of the Memorandum in defining open data and using open licenses are laudable, the implementation does not meet the 8 Principles requirements of open government data, at least under the usual understanding of "open license," and the use of the definition to promote open data is very limited.
>
> PS. As Derek Willis points out over Twitter, the "mosaic effect" paragraphs in the memorandum are also somewhat concerning. The mosaic effect is hard to quantify and therefore difficult to limit, and this creates a big hole for keeping data government out of public reach.
>
>
>
> --
> You received this message because you are subscribed to the Google Groups "Open House Project" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to openhouseproje...@googlegroups.com <mailto:openhouseproject%2Bunsu...@googlegroups.com> .
> To post to this group, send email to openhous...@googlegroups.com.
> Visit this group at http://groups.google.com/group/openhouseproject?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>
>
> --
> You received this message because you are subscribed to the Google Groups "Open House Project" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to openhouseproje...@googlegroups.com <mailto:openhouseproject%2Bunsu...@googlegroups.com> .
> To post to this group, send email to openhous...@googlegroups.com.
> Visit this group at http://groups.google.com/group/openhouseproject?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
--
You received this message because you are subscribed to the Google Groups "Open House Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openhouseproje...@googlegroups.com <mailto:openhouseproject%2Bunsu...@googlegroups.com> .
To post to this group, send email to openhous...@googlegroups.com.
Visit this group at http://groups.google.com/group/openhouseproject?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
Combined Federal Campaign #10201
--
You received this message because you are subscribed to the Google Groups "Open House Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openhouseproje...@googlegroups.com <mailto:openhouseproject%2Bunsu...@googlegroups.com> .
Josh Tauberer
Not all licenses that apply to gov data are bad.
Hi, Bonnie.
Sure. All things being equal, open is better than closed.
But the memorandum is specifying baseline requirements that would apply not only to medical research data but also to core government information, such as... the law itself, administrative records (think Benghazi email thread), financial disclosure records (STOCK Act), government spending data, and so on.
Just think if the Administration released the Benghazi emails under an "open license" that required attribution. The DOJ might be filing a suit with ABC now because they don't like how ABC reported on the emails, and they'd claim ABC failed to include proper attribution. In light of the AP phone records snooping, this is the last thing we need.
So while the gov't can't put a copyright license on content it doesn't have copyright in to begin with, your point that "license" doesn't necessarily mean a copyright license is exactly the root of my concern.
We can't have transparency if accessing the information comes with restrictions on speech. That's why "license-free", and not "open license," is the right policy when we're setting general baselines for open government data.
In rereading this thread, it seems to me that we need to clarify the definition of license. Licenses are not necessarily about copyrights, Generally a license is a contract or arrangement in which a property owner permits certain uses of their property by a third party. The property can be tangible (a physical thing, like a DVD), intangible property (intellectual content protected by patent, copyright, trademarks or trade secrets) or both. Government attorneys always emphasize that in contracting, program managers must be careful to specify the deliverable (data in a physical format) which gets you at least one copy (or number specified) AND must also specify the appropriate data rights. Without the data rights (or license), you might not be able to do anything much with that deliverable except to read it or use it as wallpaper (first sale). Not all licenses that apply to gov data are bad. Take a look at NLM license agreement for use of the visible human dataset -- which in esssence reinforces that the data is and will remain free of copyright or any other claims by third parties. http://www.nlm.nih.gov/research/visible/getting_data.htmlBonnie Klein
--
You received this message because you are subscribed to the Google Groups "Open House Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openhouseproje...@googlegroups.com.