Hi Charlie and Greg,
Good discussion, and I'm glad that Charlie pointed out the need for an
"open architecture". This is also something based on very recent
experience with Recovery.gov related disclosures.
We’ve started to work with the real data published as part of the
American Recovery and Reinvestment Act (ARRA; aka the “stimulus
package”). Data formats are obviousless important - there’s simply too
much information to effectively monitor and use unless it comes in
formats that lend themselves to aggregation and analysis. The
architecture of data dissemination is also a vitally important aspect
of any transparency or publication measure, but is more poorly
understood and has received less recognition than formats. If you
can’t get data from clear, easy to find, and easy to use services,
disclosure is pretty meaningless.
That’s why we were so excited to learn that OMB was requiring agencies
to publish feeds.Feeds (or rather Atom feeds to be specific) are a
wonderful and convenient method. They lend themselves to distributed
(and hopefully robust) publishing scenarios, and have the advantage of
being very widely supported, flexible, extensible, and easy to use.
However, we’ve managed to find only 25 or so feeds published by
agencies participating in the stimulus. Feed discovery is a major
issue that needs to be ironed out. It’s also likely that not that many
agencies are yet in compliance with OMB’s Feb 18th guidelines for
stimulus disclosures. To hazard a guess, it seems that the federal
government’s existing IT infrastructure is not very well equipped to
“do transparency”.
Different agencies are probably mainly sending their stimulus reports
as emails with Excel spreadsheets attached for publication at
Recovery.gov. While this ad hoc solution probably works OK, it is
pretty depressing that (as Charlie mentions) many millions of dollars
of IT infrastructure investment in agency systems can not be applied
for something like this. So, for the interim, the most comprehensive
source of stimulus disclosure data is at the Recovery.gov site.
Ironically, Recovery.gov site does not publish its own feeds of the
data obtained (emailed?) from different agencies. Because Recovery.gov
doesn’t have any convenient feeds pointing to their more comprehensive
collection of disclosure reports, I’ve just spent several hours
writing a script to “scrape” the Recovery.gov site in order to mine it
for all available Excel weekly report spreadsheets.This is not an
optimal solution, since scrapping tends to break if Recovery.gov makes
even minor changes to its styling / layouts. Feeds would be much more
reliable to identify disclosure related resources.
If you want to see a comparison of data obtained from agency feeds and
data obtained from scrapping Recovery.gov take a look at Erik Wilde’s
blog post:
http://dret.typepad.com/dretblog/2009/03/recovery-reporting-report.html
and this page that visualizes these different data sources on a
timeline:
http://isd.ischool.berkeley.edu/stimulus/feeds/site-vs-feeds.html
All of this goes to show that we there needs to be much more progress
on following through with the stimulus transparency measures. But this
exercise also shows how useful feeds (especially Atom feeds) can be
for disclosure. They offer a simple solution to reliably get published
resources of disclosure data, and unlike scrappers, they require no
custom coding and are not vulnerable to style changes on web pages. If
more agencies published easy to discover Atom feeds, civil society
groups and even Recovery.gov would have a simple and reliable way to
get comprehensive accounting for $800 billion in spending.
Best!
-Eric
On Mar 30, 1:38 pm, Greg Elin <
ge...@sunlightfoundation.com> wrote:
> Hi Charlie,
>
> Thanks for these questions. I've replied inline as best I can. But the gist
> of what I say is this: the openness is happening and we need to push it. Can
> you help put forward ideas on doing this correctly? (Respond on list or
> off.)
>
>
>
> On Fri, Mar 27, 2009 at 10:43 AM, Charlie <
synech...@gmail.com> wrote:
>
> > Hello All:
>
> > Last week Vivek Kundra, the new Federal CIO, said that he is directing
> > Federal agencies to build transparency, participation and
> > collaboration into the architecture of Federal agencies. I think you'd
> > get the idea by watching Kundra's talk here
>
> >
http://www.americanprogress.org/events/2009/03/doors.html
>
> > Given Sunlight's role to use the revolutionary power of the Internet
> > to make information about the Federal government more meaningfully
> > accessible to citizens, the questions I'll pose to the list are as
> > follows:
>
> > a) Whether this architecture should be determined behind the closed
> > doors of the government, such as its Architecture and Infrastructure
> > Committee, or whether this architecture should be developed in the
> > open where citizens can witness its development?
>
> The conversation about
data.gov and architecture should absolutely follow
> the principles Obama Open Government
> Memo<
http://www.whitehouse.gov/the_press_office/TransparencyandOpenGovernm...>.
> Also, we all need to be vigilant to insure that result. There is every
> indication that conversations about specific *details *on this list and
> elsewhere on the web are being seen and taken into consideration. And
> meetings are talking place, such as the meeting from the video to which you
> linked.
>
> The Administration--any Administration--faces a problem with regard to
> actively responding to conversations: informal conversation in public forums
> is frequently interpreted as policy. This problem...transitioning from
> formal hearings and rule commenting and behind closed door lobbying to the
> openness and informal continuous interactions of the web...is what the Open
> Government Directive speaks. Hopefully.
>
> Everyone is learning right now. There are no Government 2.0 experts.
> Everyone needs to participate and yet it is hard to get certain work done in
> large groups. So expect all sorts of conversations. And the best we can do
> is to charge ahead with the conversations and not wait to be asked.
>
>
>
> > b) Whether this architecture should be a government purposed or an
> > Internet-based architecture?
>
> It's hard for me, personally, to see the any new openness coming from these
> players not being based on the Internet. Vivek is 34 yrs old and turned
> toward the web for data feeds at DC Octo. Beth
> Noveck<
http://www.nyls.edu/faculty/faculty_profiles/beth_simone_noveck>,
> at the Whitehouse OSTP working on the Open Govt Directive, has written about
> Wiki-Government<
http://www.amazon.com/Wiki-Government-Technology-Democracy-Stronger/d...>.
> I could go on. (But again, we have to keep the pressure on.)
>
> > Surely, behind the closed doors of the government Federal agencies
> > spend many millions of dollars each year on Enterprise Architecture
> > activities, yet there's no evidence of an outcome. Where information
> > on them is available, the architectures themselves appear rudimentary
> > at best and the problems on which they focus show no evidence of value
> > to citizens. While at the same time the World Wide Web Consortium
> > (W3C) defines a set of open standards through which innovators across
> > the globe continue to outpace the government despite significant
> > electronic government spending. Consider the effectiveness of OpenID
> > v. the government's E-Authentication initiative.
>
> I've been learning a bit about the Enterprise Architecture activies and the
> more I learn the more it seems to have the same strengths and weakness of
> large scale IT normalization projects. The baby is the fact government has
> been working on e-gov for a while and organizations are talking. The
> bathwater is difficulty in experimenting inside government and that
> Enterprise Architecture is still, IMHO, a developing field. (Heh, so the
> govt is experimenting in a sense...)
>
> > c) Shouldn't there be a partnership for Sunlight and W3C in defining
> > an open architecture for an open government? The government seems to
> > get stuck on issues that W3C has been very effective at solving, like
> > information sharing.
>
> John Wonderlich and I are interacting with W3C's e-gov working
> group<
http://www.w3.org/2007/eGov/>.
> The definition of an open architecture is hard work and I hope you will
> participate. It will take time.
>
> > I suspect that the bureaucracy that continually fails at keeping pace
> > with society's ability to innovate needs your help in creating an open
> > architecture for an open government.
>
> You are too kind! Sunlight's success is because we practice crowd sourcing
> and tap the thinking of people like yourself. The openness meme has caught
> and is BIT. WE NEED YOUR HELP!
>
> --
>
> Greg Elin
> Chief Evangelist for Sunlight Foundation (
http://sunlightfoundation.com)
> Sunlight Labs (
http://sunlightlabs.com)
>
ge...@sunlightfoundation.com
> g...@fotonotes.nethttp://
twitter.com/gregelin