Re: [sunlightlabs] Sarah's inbox

[Pedro Markun]

Discussão na lista da Sunlight sobre os emails recuperados da Sarah Pailin.

Vale ver o processo deles (e checar o Elena's Inbox pra quem não viu, que é genio!)

Não acompanhei o caso, mas fiquei intrigado com a possibilidade de usar o FOIA para recuperar emails de figuras públicas. Isso valeria dentro da nossa lei de acesso?

No encontro com o Paulo Pimenta propus isso como uma das possíveis diretriizes de um 'gabinete aberto'... seria legal ter uma ferramenta dessas para esse caso.

abs,
Pedro Markun

On Sat, Jun 11, 2011 at 11:53 AM, Tom Lee <tl...@sunlightfoundation.com> wrote:

To answer Paul's question: I have no idea if anyone will be standing up the Elena's Inbox code to this end. It's open source (https://github.com/sunlightlabs/elenas_inbox) and I'd be thrilled if someone did.  But at Sunlight we decided not to pursue this, for a couple of reasons.  First, I've read that our friends at Pro Publica are working on an interface for the email; it's their FOIA (along with MoJo and MSNBC), and I wouldn't want to try to steal their thunder.  Second, while there's a decent chance that these emails will be relevant to the upcoming cycle in a substantive way, at the moment I think it's safe to file them under the heading of "neverending Palin sideshow."  I personally find it fascinating, but it's not exactly within our mission.

Still, I'm hard pressed to resist the temptation to hack away at these things a bit, and would be glad to see collaboration toward that end on this list -- at the very least I've got it in my head to try to throw together a fun little infographic.

The link Luigi gave is a great start.  If anyone's got a bulk link, even better -- I should be able to supply one in a few hours.  

The real meat of the Elena's Inbox code is a bunch of awful regex work, very little of which will probably be applicable to this dataset.  But I think that's what needs to be done: parsing these blobs of text (which are organized by day, apparently) into discrete objects with identified senders.  The workflow goes something like this:

1. Segment emails into separate objects

2. Parse emails to create objects representing senders and recipients

3. Merge those person objects (many of which will be duplicates) based on some combination of heuristics and manual cleanup

4. (optionally) Build collections of emails into threads using an algorithm that looks at subject and date

At this point I honestly can't even remember the order in which the EI management commands need to be run to do the import.  But I'll have a look at forking the code and adapting it to the intake task.  At a glance it appears that these emails are a *much* cleaner dataset than the Kagan release, but it'll be hard to say for sure until I've dug in.

The sender merge is the biggest PITA -- lots of need for cleaning up OCR errors, variations in the "from" field based on different email clients being used, etc.  I imagine there's a super-slick way to employ Google Refine to share this work, but I'm not familiar enough with that tool to undertake that task myself -- pointers welcome!

On Sat, Jun 11, 2011 at 9:16 AM, Luigi Montanez <luigi.m...@gmail.com> wrote:

The NY Times has scanned all of the emails into Document Cloud, making them available as PDF and plain text:

http://projects.nytimes.com/palin-emails

Not quite the same interface as Elena's Inbox, but the most important bit -- search -- works well.

On Sat, Jun 11, 2011 at 1:31 AM, Paul Schreiber <paulsc...@gmail.com> wrote:

Is anyone going to do a redux of “Elena’s inbox” with the Palin emails?


still facepalming,
Paul

--
You received this message because you are subscribed to the Google Groups "sunlightlabs" group.
To post to this group, send email to sunlig...@googlegroups.com.
To unsubscribe from this group, send email to sunlightlabs...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/sunlightlabs?hl=en.

--
You received this message because you are subscribed to the Google Groups "sunlightlabs" group.
To post to this group, send email to sunlig...@googlegroups.com.
To unsubscribe from this group, send email to sunlightlabs...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/sunlightlabs?hl=en.

[Fabiano Angélico]

Markun: em tese, sim, valeria.

Na Inglaterra também teve uma ótima história sobre emails de autoridades. Com base no FOIA, um jornalista pediu acesso a todos os emails que contivessem a palavra "embarassing":
http://www.bbc.co.uk/blogs/opensecrets/2010/06/embarrassing_emails.html

 

Fabiano Angélico
Twitter: @fangelico
Blog: Notas Soltas (http://algumasnotassoltas.wordpress.com)
Facebook: fabiano.angelico
Skype: fabiano.angelico
(55) 11-8270.3020

---

De: Pedro Markun <pe...@esfera.mobi>
Para: thac...@googlegroups.com
Enviadas: Segunda-feira, 13 de Junho de 2011 9:35:20
Assunto: [thackday] Re: [sunlightlabs] Sarah's inbox

--
Você está recebendo esta mensagem porque se inscreveu no grupo "Transparência Hacker" dos Grupos do Google.
Para postar neste grupo, envie um e-mail para thac...@googlegroups.com.
Para cancelar a inscrição nesse grupo, envie um e-mail para thackday+u...@googlegroups.com.
Para obter mais opções, visite esse grupo em http://groups.google.com/group/thackday?hl=pt-BR.