[OTT #81] the REAL costs of a sortable, searchable website for state transparency

[Jon Bartholomew]

Hi all,

I’d like to propose a conference call to discuss something.

In 2009, we proposed that Oregon create a state transparency website that has a searchable, sortable interface. However, that specific requirement got taken out of the legislation because the Department of Administrative Services said it would cost $7 million to do. So instead, currently at www.oregon.gov/transparency, you can get raw data in spreadsheets. You can not do a search of the data, and databases are not linked together.

 

Now, the reason they put that number on the cost is that they say to do this, they need to spend the money to get all the state’s data collection standardized. They are supposedly working toward that goal slowly, but for now, all the data lacks consistency. However, as you can see, there is at least some raw data available, and one of the folks emailed here, Ersun Warncke took this data and made something really cool at http://transparency.salem-news.com/

 

So what I want to discuss is this – what would it REALLY cost to do what we think needs to be done? And by what we think needs to be done, I mean being able to search the website for something like “Intel” and be able to view all the contracts with the state Intel has, what tax subsidies they receive, what they and their executives have given in campaign contributions, etc.

 

Is there any interest in having a conference call about this?

 

Thanks.

 

 

..Jon

 

======================

Jon Bartholomew

Policy Advocate

OSPIRG - www.ospirg.org

jo...@ospirg.org

cell 503.358.3833

office 503.231.4181 x374

twitter http://twitter.com/OSPIRGJon

facebook http://www.facebook.com/OSPIRG

 

--
You received this message because you are subscribed to the Google Groups "Oregon Transparency Taskforce" group.
To post to this group, send email to oregon-transpa...@googlegroups.com.
To unsubscribe from this group, send email to oregon-transparency-...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/oregon-transparency-taskforce?hl=en.

[Pete Forsyth]

Jon,

Thanks -- I'm really excited to see you convening this discussion, and I hope you have some takers soon -- there are some great folks on this list who can probably offer some great insights into what it would take.

As far as I'm concerned (and I want to credit Kristin with articulating this the most clearly), the greatest opportunity as far as cost is in getting government agencies and DAS all working on the same system, which outputs searchable data to the public as one of its many features. If the system were redesigned to work that way (which is pretty standard in large organizations these days), the cost savings in intra-government efficiencies should vastly outweigh any costs associated with the public-facing side. (Of course, there are costs associated with a transition like this, and it's impossible to comment on those without first gaining a pretty sophisticated understanding of how things currently work.)

And as a related point, the biggest fear is that there may be efforts to upgrade computer systems within the capitol, which address some of these needs but do NOT address the issue of transparency to the citizens of Oregon. If there is a rollout of a new computer system in the next few years which happens independent of this move toward transparency, that will be truly tragic, and I think that's the thing many people on this list would most want to avoid.

On a separate note, I would like to direct people's attention to the book Full Disclosure, published in 2007 by Cambridge University Press, written by Archon Fung, Mary Graham, and David Weil of Harvard. The book reflects a rigorous and practical examination of government transparency efforts. If anybody has thoughts, comments, or questions, please send them my way, as I am working pretty closely with two of the authors, and I'm sure they would be interested in what is developing in Oregon.

-Pete

[Jon Bartholomew]

Thanks for responding, Pete.

And sorry I have been so slow to follow up. It’s been crazy here. Perhaps we don’t need a conference call as much as having an email discussion.

I like the idea of having the state or even someone line Sunlight Foundation fund a contest like Apps for America. Of course, it would be Apps for Oregon! Perhaps in the process of having people develop apps, we can identify where the gaps are that need to be filled in terms of what information is available and what needs to be made more consistent.

 

I totally agree with your concerns about computer system upgrades being done without an eye to transparency. It’s going to require pressure on agencies (particularly DAS) to keep it as a top priority, and also leadership, so we should do what we can to make this an issue in the gubernatorial race. The next governor needs to be clear how he wants things to go.

 

Ersun’s comments to me show there is a lot of work that still needs to be done just to get agencies to understand that their data is public information! At least currently the AG is going around the state collecting comments on improving overall transparency. But putting it online is more of a job for the transparency advisory commission.

 

By the way, the next Transparency Commission meeting will be on May 27^th from 9 to 11 at the state capitol in hearing room F. We don’t have an agenda finalized yet, but I have been told we will have 20-30 minutes set aside for public comment. And of course, we will accept written testimony as well.

 

So let me know if you are going to be able to come to that meeting (and Ben, let’s get together before hand).

 

..jon

 

 

---

[Jeff Schwaber]

On Mon, May 3, 2010 at 3:44 PM, Jon Bartholomew <jo...@ospirg.org> wrote:

Hi all,

I’d like to propose a conference call to discuss something.

In 2009, we proposed that Oregon create a state transparency website that has a searchable, sortable interface. However, that specific requirement got taken out of the legislation because the Department of Administrative Services said it would cost $7 million to do. So instead, currently at www.oregon.gov/transparency, you can get raw data in spreadsheets. You can not do a search of the data, and databases are not linked together.

 

Now, the reason they put that number on the cost is that they say to do this, they need to spend the money to get all the state’s data collection standardized. They are supposedly working toward that goal slowly, but for now, all the data lacks consistency. However, as you can see, there is at least some raw data available, and one of the folks emailed here, Ersun Warncke took this data and made something really cool at http://transparency.salem-news.com/

 

So what I want to discuss is this – what would it REALLY cost to do what we think needs to be done? And by what we think needs to be done, I mean being able to search the website for something like “Intel” and be able to view all the contracts with the state Intel has, what tax subsidies they receive, what they and their executives have given in campaign contributions, etc.

Jon,

It's very, very difficult to accurately estimate this sort of project. The number of datasets, the amount of normalization, the amount of metadata that needs to be generated, the number of ways that people want to search and, probably most importantly, figuring out what people actually want to do with the site. $7mil is a lot, though. It's far more likely that that estimate is talking about the cost of pushing the data normalizations back through the system, rather than maintaining a translation layer. 

Generally I'd say that the easiest way to find out would be to find the people who set up the current website and a couple of outside experts and ask them all how long the ideal would take. But you can't really find that out until you have a proposal for what the ideal website is: just as an example, when you say search for Intel and see contracts with Intel, you mean not that we should search for the keyword intel and see what documents have that keyword, but that we should be able to search for the _entity_ intel and have things relating to the _entity_ appear. This is a much more complicated process than keyword searching.

The best way costs money. It's to hire a UX designer to put together a prototype and then hire an agile team[1] to work on it for two iterations. That'll cost $5-10k but it'll give you a pretty good estimate of what you can get done for how much.

Thanks,

Jeff

1. Yes, of course I do this for a living and would be happy to take on such a task. =)