scraped data - could you run an apps contest on it?

[John Geraci]

Hey all,

So the idea for an Apps for Everywhere contest is taking shape, as a
result of conversations with various people, most of whom are on this
list.

Here's an update on the idea that I posted to DIYcity the other day:
http://diycity.org/diycity-main-group/update-apps-everywhere-idea

At the end of that post, I distilled the idea as it stands into bullet
points:

• host an apps contest for the whole world
• that taps into local developer networks in cities to drive
participation
• that relies on state-of-the-art scraping where necessary to build
apps
• but that also (and maybe more importantly) acts as a stimulus for
cities everywhere to open up their data and make it available to
developers to build on.

Lots more to it, of course, but those are the big, salient points.

The first big question in my mind with this idea is: could you
actually hold a contest where people built apps almost entirely on
scraped data? What would the quality of apps be like? What would
shelf life of such apps be? Would it be realistic to ask developers
to scrape their own data, or would we need to somehow scrape and
organize data for them beforehand, and then just point them to it?

I'd love to hear from any developers on the list who have opinions
about these things, as well as ideas on how to make scraping work.

John

[John Geraci]

Anyone have any thoughts on this?

Apps built on scraped data - interesting and useful? Cheap and
limited in effectiveness? Difficult/easy to envision? Difficult/easy
to execute?

-j

[Dan Knauss]

It probably varies from case to case. Locally I know we can get a lot of data by email and build from that. Spotcrime already scrapes the police and fire stuff, and they have an unpublished but open API.

Dan

Sent from my iPhone

[Dan Knauss]

Given some recent events that have really brought the data access/lack of access problem to a head in Milwaukee with regard to 911 dispatch logs and crime data, I think there is going to be an effort to build some independent scraper-based applications. We need to a public information void created by the move to an encrypted IP-based radio system. Journalists and politicians remain hesitant to specify and push for solutions as our police department openly flaunts the lack of access as a tool for information suppression. 

I've posted more about this at DIY city and the RJI News Collaboratory.

http://diycity.org/diy-milwaukee/questions-journalists-politicians-and-police-department-about-public-information-acces

[John Geraci]

I was out for beers last night with David Eaves, the person who organized last weekend's Global Open Hackathon, and he mentioned a site called Scraper Wiki (http://scraperwiki.com/).  It's a site that is organizing and structuring the scraping of public data into something that is manageable by anyone (and can be contributed to by all).

Sounds very cool though I haven't had a chance to look too closely at it yet.  It may be helpful for you with your 911 problem in Milwaukee.

Also sounds like something we would definitely want to tie in to any sort of Apps for Everywhere contest.  A contest built on top of a wiki about data scraping all over the world - that is starting to sound increasingly cool, increasingly do-able, and increasingly valuable to the world community of civic developers.

David really liked the idea of the contest, btw, so we may be able to work together with the Global Open people, and their constituent groups, to get it more traction if we decide to give this a green light. 

Though he felt pretty strongly that any contest should NOT have monetary prizes involved, but should instead be just about developers in cities trying to out-do each other and make their city the best.  Still mulling that over...

-j.

[@marianogoren]

Here in Buenos Aires, Argentina, the government does not agree in our
perspective of open data. I've had a meeting with some representants
of the digital reporting system of the city. The idea was to introduce
seeclickfix, but their demand was a system with limited functionality
and closed. I think that the confrontative state of politics in here
makes the project unviable.

Maybe we should think in other way of work with third world cities,
but I can't figure exactly how =)

cheers,
MAG
www.marianogoren.com

[John Geraci]

Mariano - I think your input here actually makes a strong case for
doing a contest focused on scraped data, as opposed to looking for
gov's to open up their data.

My feeling is that if we were to hold such a contest, scrape the data,
throw it into Scraper Wiki, and build apps on that, that could really
highlight the utility of these kinds of apps/products and move the
ball forward for opening up data in cities like Buenos Aires.

As things stand right now, the govt there is indifferent/hostile
toward opening up data. A contest based on scraped data could
generate a lot of local interest in opening up data, and might start a
conversation there about the benefits of open data. At least amongst
the public, if not amongst the government. But that's the first step.

The big question for me is not "is the government open to these kinds
of things" but are the coders in these cities ready to focus on this?
Is there an awareness among local developers/hackers about this kind
of thing, and would they participate in a contest, or would they be
indifferent?

David E. said he thinks developers around the world will be ready in a
year. I believe that if you throw a big contest and generate a lot of
noise about it, you can accelerate that process, making everyone more
aware about civic hacking, and get them ready in a couple of months.

[Dan Knauss]

Nice! This looks useful.

I think our PD will be pressed to provide a webfeed of the dispatch
audio. That's pretty common. It ought to accompany, not replace, the
text output from their dispatch software.

Dan