Episodic Discussion - Open Contracting Data Standard Use cases for the Media

[Michael Roberts]

Hello,

On Friday May 16th, we had our first community web meeting on media use cases. Below is a summary of the discussion. The next web meeting will be announced in the coming days. Over the next week, we would like encourage further comments on media use cases for an open contracting data standard. Specifically, we would like to hear your thoughts on the meeting summary and/or on the following three questions:

1) How does the media make use of public contracting information right now?

2) What would happen if the media had better access to public contracting information in a machine readable format?

3) What media use cases might be priorities for an open contracting data standard?

Please introduce yourself (name, organization, interest in the open contracting data standard) when posting your comments.

At the end of the week, we will summarize these comments.

Cheers,

Michael

——

Open Contracting Data Standards

Community Web Meeting - Media Use Cases

Public contracts can be found in the media in many ways: from tender notices in newspapers, to investigative reporting that digs into cases of corruption, overspend or poor-management of public finances. Journalists may turn to contracting information to inform existing stories, or might start looking in contract data to find the stories hidden within it.

With more and more data becoming available, the need for media may appear to be growing - and yet the resources available for investigative reporting are on the decline. In this Web Meeting we will explore how the media are using data on contracts, and how they could use data in future - identifying key use cases for an open contracting data standard.

Community Web Meeting Summary (Held Friday May 16, 2014, 8am EST)

Presenter: Roberto Rocha: Interactives and data editor, The Montreal Gazette

Challenges facing investigative journalists in Montreal relating to contract data:

1)  Finding good, reliable data.

• Journalists sometimes produce their own data either by scraping online databases or by using API’s. (Example Montreal Gazette - documents.montrealgazette.com).
• Sometime you need to be a skilled programmer to get data from databases that are hard to access.
• The journalist also needs to understand if the data was released for political motivations, who put the data together, and what might be missing.

2) Contract and tendering systems outdated (in Montreal)

• Simple to bribe bureaucrats and engineers.
• The system has traditionally been very opaque in terms of disclosing contract information.
• Journalists need to spend weeks at city hall looking through documents, making connections, and finding anomalies.

3) Cleaner, more standardized data

• Often 80% of the time is spent cleaning up ‘dirty’ data and getting it into a format for which it can be analyzed instead of spending that time telling the story.  (Note: esp. important given shrinking North American media budgets).
• Still the norm to have documents shared in PDF rather than as machine-readable data.
• Reporters at the Montreal Gazette often still use old methods to find contracting-related stories since its the only way to investigate public contracts in Montreal.

Presenter: Friedrich Lindenburg: Coder, developer. Digging Deeper with Data Journalism

1) What can done to allow journalists to look at the data themselves? (In Friedrich’s experience one cannot expect journalists for have much more technical knowledge than using Excel)

• Putting the data out in forms journalists can use it and are familiar with like Excel.
• Creating simple tools for journalists to use.

2) What are some journalistic contracting questions relating to Open Tenders Electronic Daily (TED)? (A portal is designed to  create a join European market for government services and works)

• What are the biggest suppliers in my city, state, country?
• What are the biggest authorities in terms of contracting?
• Which industries are taking up a lot of money?
• Is there a lot of defence spending or a lot of infrastructure spending?

• In terms of asking these questions the most interesting bit is not the tender announcement (which TED was made for), but instead the contracting awards.

3) What is needed to improve Open TED and its use of this contracting data?

• Provide training for those might need it.
• Data quality improvements such as:
• 5% of procurement documents received as paper.
• Companies names entered as a string.
• Authority identities needed just like Open Corporates is doing for corporate identifiers - perhaps http://publicbodies.org/  (ie asked to classify themselves such as national government, regional government, or public utilities, and many are incorrect.
• For ⅓ of all tenders, the contract award is not published.
• Many more examples.
• ‘Blaming and shaming’ site.  A site that shows what authorities have missing fields, or who have no published awards, or who have not published certain feilds etc. In many cases laws exist already on what data needs to be provided publicly. The European procurement office can’t monitor this and won’t do this. Thus an external office/body is needed.
• Bridging journalists, government, technologists. The Data Harvest conference http://www.journalismfund.eu/dataharvest14  brings together journalists from all EU states. Friedrich has been working to bridge different groups such as the European Publications Office, companies that can analyze data, and journalists.

Comments: Summarized key participant comments regarding improving the publication and use of open contracting data

Lisette Garcia - Director FOIA Resource Centre, U.S.A. - former news reporter and lawyer - https://twitter.com/foiafactor

Understanding and promoting the legal statutes that already exist:

• Journalists often may not know how to use the tools (scrapers tools etc.) but she thinks its also the case that often journalists are not aware of what laws are on our side. In addition, she thinks the document issuers may not be aware of the laws for publishing data in various formats that they are required to do.
• Useful to get all the relevant parties together (legal, technical, advocates etc.)
• Why not use the big stick (laws that currently exist) that is out there. Lisette noted that the Data Coalition (trade group) is looking at agencies that agreed to publish online in standardized formats   
• Laws do not specify fields but sometimes recommend standards such as XBRL
• It would be useful to make recommendations at the field level

Friedrich Lindenburg - Open TED

• Journalists want more data regarding:
• people who have put in offers but have not won - not aware of anyone publishing this type of data; and
• to answer certain questions you can only do this with the actual contract documents.  (Questions such as: What services were to be provided under what timelines?)

Jose Alonso - Wed Foundation

• Concurs with Lisette re: laws: Jose had a related chat with a person on transparency and environmental resource issues in Europe. This person felt the open data community lacked legal expertise - laws, regulations, and all sorts of different instruments that when combined properly could help the community to push the government to release the things that community really wanted.

Juan - Programmer Paraguay

• He is working with contracting data (Json-LD) with the government in Paraguay. It is hard to find a model they could use with the phases of the contracting process. He stressed the need for a project such as the Open Contracting data Standard and how the same model could be then used for other systems from around the world. He is keen to use this standard.

Tim Davies - Web Foundation

• Many contracts are often only published over a certain threshold making it hard to aggregate if lots of the spending is under the threshold.

[end of web meeting]

---

Michael Roberts

Open Data Technical Manager

+1 514 802 9528
@michaeloroberts | mich...@webfoundation.org
World Wide Web Foundation | 1110 Vermont Ave NW, Suite 500, Washington DC, 20005, USA | www.webfoundation.org | Twitter: @webfoundation

[Lisette Garcia, FOIA Resource Center]

Thank so much organizers & fellow participants! The followup below, in support of the points raises during our first session last week, was offered by Peter Warren, policy director for Congressman Darrell Issa, Chairman of the U.S. House Oversight & Government Reform Committee:

"One of my first jobs in Washington, DC was helping get a sort of think tank off the ground – known as STATS -- that was created to help journalists make better use of data – primarily public policy data. And the focus at the time was less on getting better data than improving the data literacy of journalists. STATS still exists. Link is: http://stats.org/about.htm"

 

I also renew last week's invitation for your feedback on the GRANTS bill, designed to improve transparency of the fastest-growing sector of U.S. government spending:

http://www.gpo.gov/fdsys/pkg/BILLS-113hr3316ih/pdf/BILLS-113hr3316ih.pdf

Finally, here is a summary of the DATA Act, America's first open data law, the passage of which Data Transparency Coalition recently secured:

http://us5.campaign-archive1.com/?u=1fe27980fc8b5dda1bc362b91&id=5e1167e725&e=b60d553781

Thank you! Excited about the valuable difference this group is sure to make. --Lisette (202) 413-5243 @foiafactor

--
You received this message because you are subscribed to the Google Groups "Public OCDS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to public-ocds...@webfoundation.org.
To post to this group, send email to publi...@webfoundation.org.
Visit this group at http://groups.google.com/a/webfoundation.org/group/public-ocds/.
To view this discussion on the web visit https://groups.google.com/a/webfoundation.org/d/msgid/public-ocds/88B7EF36-6219-43B5-923C-C22A3D2BA040%40webfoundation.org.

[Michael Roberts]

Hi Lisette,

Thanks for sending these followup links.

"One of my first jobs in Washington, DC was helping get a sort of think tank off the ground – known as STATS -- that was created to help journalists make better use of data – primarily public policy data. And the focus at the time was less on getting better data than improving the data literacy of journalists. STATS still exists. Link is: http://stats.org/about.htm »

I think you made a strong case for journalists and the open data advocate community itself to understand the laws that already exist in various countries. As mentioned in the web meeting last week

we are encouraging governments to commit to open contracting through multistakeholder initiatives like the OGP in order to build a global norm and international community of practice.

However I think its also useful to be aware that in many cases public contract publication laws (sometimes including machine readable data) already exist but are not being followed. Has anyone begun to compile legal compliance gaps around open data that might help to inform both journalists and the broader community?

Cheers,

Michael

---

Michael Roberts

Open Data Technical Manager

@michaeloroberts | mich...@webfoundation.org
World Wide Web Foundation | 1110 Vermont Ave NW, Suite 500, Washington DC, 20005, USA | www.webfoundation.org | Twitter: @webfoundation

To view this discussion on the web visit https://groups.google.com/a/webfoundation.org/d/msgid/public-ocds/19476C08-E2FF-4F72-B4B0-B8323561C1E2%40gmail.com.

[Lisette Garcia, FOIA Resource Center]

Happy to help! I wasn't sure exactly what you mean by this and what the scope might be:

"compile legal compliance gaps around open data that might help to inform both journalists and the broader community"

If stakeholders lack awareness of the laws in their favor, why would we start by chronicling gaps? And do we mean state by state, country by country, everything?

This nonprofit has compiled a list of relevant laws: http://www.nfoic.org
But my experience has been -- even on the NICAR listserv -- that reporters are less interested in learning what pertinent laws allow than just getting a yes or no in requesting access to a record.

[lmarch...@worldbank.org]

Perhaps one issue that would be interesting to explore, building on Lisette and Michael's points - is the role of oversight authorities in ensuring compliance with transparency measures. What are the incentives that have worked best in order to maximize compliance with obligations to publish data and documents?  If there are those on the list with experience that could speak to this issue, it would be very interesting for the standards team.



"Lisette Garcia, FOIA Resource Center" ---05/21/2014 06:38:13 PM---Happy to help! I wasn't sure exactly what you mean by this and what the scope might be: "compile leg

From: "Lisette Garcia, FOIA Resource Center" <lisette....@gmail.com>
To: "publi...@webfoundation.org" <publi...@webfoundation.org>
Date: 05/21/2014 06:38 PM
Subject: Re: [public-ocds] Summary & followup: Open Contracting Data Standards
Sent by: publi...@webfoundation.org
      

---

--
You received this message because you are subscribed to the Google Groups "Public OCDS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to public-ocds...@webfoundation.org.
To post to this group, send email to publi...@webfoundation.org.
Visit this group at

http://groups.google.com/a/webfoundation.org/group/public-ocds/.
To view this discussion on the web visit https://groups.google.com/a/webfoundation.org/d/msgid/public-ocds/2E221EE8-ECFE-455A-A9F0-FFC647E6F9E7%40gmail.com.

[Lisette Garcia, FOIA Resource Center]

Well, in the US (at all levels) the main incentive is to avoid lawsuit. The law provides a right of suit with fee shifting to a prevailing requester. My consulting firm is based on avoiding lawsuit altogether, though, because it is too slow and complex for the needs of breaking news.

A second layer of incentive at the federal level is an annual report to Congress and the attorney general as to the number and extent of backlogged requests. The report does not subdivide among types of data requested so delays in contract reporting, specifically, would not be immediately apparent here.

A third level also in the federal law is the supervisory office of government information services (OGIS). It is a designated mediator with nonbinding authority to settle disputes. They also are tasked with providing agencies oversight and guidance on complying more faithfully in things like contract reporting but haven't gotten around to issuing best practices in nearly a decade of existence.

A fourth level is congressional hearings where the agencies are routinely excoriated for not doing what hey must. The shaming tactic has powerful but short-term effect on behavior.

However, the best defense, in my professional experience, is for those seeking data to know what they're asking for, how to ask for it, and what they are legally entitled to receiving. When you know the law, the parties bound by it are much more likely to comply in the first instance or, failing at first, much quicker to correct.

Like I said, most reporters would like to make it the responsibility of the records custodians to inform themselves of the uppermost limits of the law and comply with it fully without the reporter being aware of what they are entitled to or following the necessary protocols to ensure these practices are adhered to.

To illustrate: yes, police should never engage in brutality. But the first check on such unlawful activity is that citizens know it is illegal and actionable in court.

On May 21, 2014, at 12:48 PM, lmarch...@worldbank.org wrote:

Perhaps one issue that would be interesting to explore, building on Lisette and Michael's points - is the role of oversight authorities in ensuring compliance with transparency measures. What are the incentives that have worked best in order to maximize compliance with obligations to publish data and documents?  If there are those on the list with experience that could speak to this issue, it would be very interesting for the standards team.

<graycol.gif>"Lisette Garcia, FOIA Resource Center" ---05/21/2014 06:38:13 PM---Happy to help! I wasn't sure exactly what you mean by this and what the scope might be: "compile leg

From: "Lisette Garcia, FOIA Resource Center" <lisette....@gmail.com>
To: "publi...@webfoundation.org" <publi...@webfoundation.org>
Date: 05/21/2014 06:38 PM
Subject: Re: [public-ocds] Summary & followup: Open Contracting Data Standards
Sent by: publi...@webfoundation.org
      

---

Happy to help! I wasn't sure exactly what you mean by this and what the scope might be:

"compile legal compliance gaps around open data that might help to inform both journalists and the broader community"

If stakeholders lack awareness of the laws in their favor, why would we start by chronicling gaps? And do we mean state by state, country by country, everything?

This nonprofit has compiled a list of relevant laws: http://www.nfoic.org
But my experience has been -- even on the NICAR listserv -- that reporters  are less interested in learning what pertinent laws allow than just getting a yes or no in requesting access to a record.

> On May 21, 2014, at 12:06 PM, Michael Roberts <mich...@webfoundation.org> wrote:
>
> Has anyone begun to compile legal compliance gaps around open data that might help to inform both journalists and the broader community

--
You received this message because you are subscribed to the Google Groups "Public OCDS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to public-ocds...@webfoundation.org.
To post to this group, send email to publi...@webfoundation.org.
Visit this group at http://groups.google.com/a/webfoundation.org/group/public-ocds/.
To view this discussion on the web visit https://groups.google.com/a/webfoundation.org/d/msgid/public-ocds/2E221EE8-ECFE-455A-A9F0-FFC647E6F9E7%40gmail.com.

--
You received this message because you are subscribed to the Google Groups "Public OCDS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to public-ocds...@webfoundation.org.
To post to this group, send email to publi...@webfoundation.org.
Visit this group at http://groups.google.com/a/webfoundation.org/group/public-ocds/.

To view this discussion on the web visit https://groups.google.com/a/webfoundation.org/d/msgid/public-ocds/OF09E8508A.E9B5EED9-ON85257CDF.005BCD6E-C1257CDF.005C511F%40worldbank.org.

[aball...@worldbank.org]

There is an Eastern European country where public sector contracts are not legally binding until they are published. I think it's Estonia, but not 100% sure.

Amparo

---

  From: lmarchessault
  Sent: 05/21/2014 06:48 PM ZE2
  To: publi...@webfoundation.org
  Subject: Re: [public-ocds] Summary & followup: Open Contracting Data Standards

Perhaps one issue that would be interesting to explore, building on Lisette and Michael's points - is the role of oversight authorities in ensuring compliance with transparency measures. What are the incentives that have worked best in order to maximize compliance with obligations to publish data and documents?  If there are those on the list with experience that could speak to this issue, it would be very interesting for the standards team.

"Lisette Garcia, FOIA Resource Center" ---05/21/2014 06:38:13 PM---Happy to help! I wasn't sure exactly what you mean by this and what the scope might be: "compile leg

[peter rabley]

Lisette

Very interesting and useful points. Having spent quite a bit of time building and extracting land records throughout the US - the other issue that always struck me were the local officials (in this case in the recorders or clerks offices) understanding the law but choosing to interpret as they saw fit  - or simply as the office had always done. In other words the procedural side of implementation that flows from the law (or in theory is supposed to) is left widely open to interpretation in practice. Not a single recording office I dealt with (more than 800) ever had published manuals etc. laying out procedures on items like access to data, recovery fee, fee for charge and in what format etc. nor worked with the county or city attorney or state attorney general to determine these. It was therefore left to firms, FOIA orgs etc. to establish them through - in many cases legal action.
Peter

To view this discussion on the web visit https://groups.google.com/a/webfoundation.org/d/msgid/public-ocds/D2E64449-1957-432E-A5B2-4E1E932E5C1A%40gmail.com.

[Lisette Garcia, FOIA Resource Center]

Sorry, this is inaccurate:

"ever had published manuals etc. laying out procedures on items like access to data, recovery fee, fee for charge and in what format etc. nor worked with the county or city attorney or state attorney general to determine these"

Just because front-line custodians were either unaware or disingenuous doesn't mean they didn't exist. Per the FOIA law itself, requesters have no legal right to have records -- including policy manuals -- created. The lawsuits only draw out recorda of policy that already exists.

[Lisette Garcia, FOIA Resource Center]

What may clarify here is that land records are typically managed by county. They are subject to state laws by incorporation. So, yes, in plenty of instances the local custodian is (sometimes willfully) ignorant of their obligations. It is, as with any interaction with a governmental body, the petitioner's burden do know what he is entitled to, not the governmental representative, which was my original point.

[peter rabley]

Lisette - thanks - perhaps I didnt frame well. It was not access to the manuals I was mentioning - but the fact that manuals dont exist. Which means staff dont know - and either are disingenuous as you mention or they simply say no or make things up. Land records are public information - and access must be provided - but it is provided in person and in hard copy only as is reflected in early state statutes (US only) - copies of hard copies are charged for as set by local regulations. Digitization of the land records (being specific) has turned all of this on its head with no clear policy around what "open access" means - and interpretations by local officials leading to things like "copy fees" being charged per record level access. Most as you are probably aware fall back to access meaning on site and through digital means. Best case some counties make freely available on their portal through search and html display, some require people to sign "valid use" certificates the approval of which falls to the clerk of the court (most VA counties). The lack of clear procedures up front allows officials to restrict access to records which are considered open for public access. I am again providing specificity to land records in the US. Peter

--
You received this message because you are subscribed to the Google Groups "Public OCDS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to public-ocds...@webfoundation.org.
To post to this group, send email to publi...@webfoundation.org.
Visit this group at http://groups.google.com/a/webfoundation.org/group/public-ocds/.

To view this discussion on the web visit https://groups.google.com/a/webfoundation.org/d/msgid/public-ocds/486AA577-2ED6-4596-98B2-3A5F3D52269A%40gmail.com.

[Lisette Garcia, FOIA Resource Center]

Hi, Peter:

I understood your original meaning. I have been pulling records, including land from recorder of deeds, since I started as a beat reporter in 1997 at Miami Herald. Except a brief period during digitization, as you mentioned, the issue was one of ignorance/willful blindness not that written policies didn't exist.

The point now, though, is that -- unless reporters are aware of the public's enforceable right of access -- we risk remaining in the same place as before despite increased facility of récords transfer in light of new technology. Newsrooms unaware of the law (incl laws requiring establishment of access procedures) take a spokesperson or records custodian's word for their nonexistence/unavailability. In-house counsel has been slow to appreciate that access, not libel or copyright, is the most significant legal issue news organizations are facing today.

I coach dozens of reporters on his stuff for a living. Knowing the law is the only difference between my clients' wins and other people's losses.

--Lisette (202) 413-5243 @foiafactor

To view this discussion on the web visit https://groups.google.com/a/webfoundation.org/d/msgid/public-ocds/CAGduYsf%2BithKF2h%3D%2B9stSgtCBKPFbmO%2BN0JL_jB9Oyoe4OYyZw%40mail.gmail.com.