Fwd: whitelist account? Re: Googler introduction

[Jeannie Stamberger]

Could someone who was working on haiti_tweeks last week, please follow-up on the request from Twitter below, and let me know that it is taken care of?
Thanks,
Jeannie

---------- Forwarded message ----------
From: Jeannie Stamberger <jeannie.s...@gmail.com>
Date: Tue, Jan 26, 2010 at 9:58 AM
Subject: Re: whitelist account? Re: Googler introduction
To: Del Harvey <d...@twitter.com>
Cc: Ryan Sarver <rsa...@twitter.com>, Brian Sutorius <bsut...@twitter.com>


Sure no problem!
Best,
Jeannie

On Tue, Jan 26, 2010 at 9:53 AM, Del Harvey <d...@twitter.com> wrote:

It would appear that the person running haiti_tweets engaged in aggressive following and accrued a number of blocks as a result, which led to the suspension.

Can you have that person submit a ticket via bit.ly/twicket so that we can discuss with them what triggered our system?

Thanks,
Del

On Tue, Jan 26, 2010 at 9:22 AM, Jeannie Stamberger <jeannie.s...@gmail.com> wrote:

Hi Ryan,
Thanks for your reply, and for the information about the best way to get faster turn-around on account help. 

I am excited to see Chirp! I am not sure if anyone working on disaster relief from Crisis Commons will be able to be there since tickets seem already sold out (Congratulations!!!).  I am sure the disaster community would love to share what we learned, if it can be of help; all of the code is open source and available through www.crisiscommons.org.

Best,
Jeannie

On Tue, Jan 26, 2010 at 9:15 AM, Ryan Sarver <rsa...@twitter.com> wrote:

Jeannie,

Apologies for just seeing this email. Del Harvey can help you with your specific requests and what we can/can't do. Also, in the future, it best to email a...@twitter.com if you are looking for a faster turn around. Those requests are ticketed and managed daily -- my inbox can suck up emails to never be seen again.

Thanks for all your work around Twitter and Haiti.

Best, Ryan

On Mon, Jan 18, 2010 at 9:44 AM, Jeannie Stamberger <jeannie.s...@gmail.com> wrote:

Hi Ryan,

Kate has informed me that a second group of developers from CrisisCamp is also working on aggregating information from the Tweak the Tweet grammar used in Haiti, using a Twitter account called #haiti_tweets.  That account has also been suspended, again probably because the search code triggered a mis-use alert.

Is it possible to get #haiti_tweets white-listed as well?
Thanks,
Jeannie

On Mon, Jan 18, 2010 at 9:37 AM, Jeannie Stamberger <jeannie.s...@gmail.com> wrote:

Hi Ryan,

It is a pleasure to meet you, and I hope Kate Starbird (the other author of this idea) get to meet you in person soon.
Thanks, Jeff for putting us in contact.

It sounds like Jeff has explained the idea behind Tweak the Tweet (a hashtag grammar showing/helping people provide essential on-the-ground information critical for relief efforts).

Due to the Haiti earthquake, we released the grammar (http://epic.cs.colorado.edu/helping_haiti_tweak_the_twe.html)  and began promoting it out of the EPIC group at University of Colorado Boulder (http://epic.cs.colorado.edu/), which has analyzed Twitter feed from past disasters.  The EPIC group is headed by Leysia Palen, who is cc'd on this email. We have been receiving traction, and people are beginning to Tweet using the grammar. 

Now we want to make sure that this information gets to the right people, and we are developing tools to search Twitter feed to find Tweets using the grammar, and rebroadcast them in a single feed.  At the CrisisCamp on Saturday, we ran into a snag where we developed search code, but the Twitter account (haititweaks) was shut down, probably because our rebroadcasting efforts triggered a mis-use alert.

Would it be possible to get the Twitter account haititweaks white-listed, so we can look at different ways that we can collect and disseminate disaster related information to aid organizations?

Thanks very much!

Best,

Jeannie

Visiting Scientist
Carnegie Mellon Silicon Valley
NASA Ames Research Campus
Moffett Field

On Jan 16, 2010, at 3:49 PM, Jeff Martin <jeffm...@google.com> wrote:

Ryan,

Given developments in Haiti and the urgent need, I'm connecting you with Jeannie, one of the brains behind TtT.

Jeannie, take it away.....

On Wed, Jan 6, 2010 at 11:26 AM, Jeff Martin <jeffm...@google.com> wrote:

Happy New Year, Ryan.

I think the near opportunity would be for you to meet with the developers of the hashtag system described in this thread, with the objective of understanding the value to disaster response, Twitter's willingness to move forward, and requirements for implementation. If interested, I'll connect you with the developers of TtT, you can schedule a meeting, and I'll try to join (I'm in Colorado but fly to CA often).

RE: RHoK, I can give you a run down of objectives if/when we meet and you can decide if it's aligned with your goals and resouces...

Best,

On Sun, Dec 27, 2009 at 8:56 PM, Ryan Sarver <rsa...@twitter.com> wrote:

Jeff,

Trying to follow up on old emails. Let me know what forms of participation you think we should consider. It's difficult for our small team to make it to too many events, but let me know how you think we could be most effective.

Best, Ryan

On Fri, Dec 4, 2009 at 5:19 PM, Jeff Martin <jeffm...@google.com> wrote:

Hey Ryan,

As Mike said, I recently helped organize Random Hacks of Kindness (RHoK), the hackathon for disaster response. One of the interesting hacks that came out of that was called "Tweaking the Tweet.  (TtT)" The general gist is that hashtags could be used to help parse and categorize the information in tweets, which would make it more analyzable and therefore more useful for disaster responders. See some snippets of conversations I've had with the TtT developers to get a better feel for it.

So I have two subjects for you to consider:

1. The next RHoK is scheduled for Feb 27 and 28. I'd like to chat with you about participation (which can take many forms)
2. If you're interested in Tweaking the Tweet, I'd like to schedule an audience with the TtT developers and your team to present and discuss.

Let me know your thoughts.

I The tag could be used for many different ways people identify locations (building names, lat long, street addresses, intersections: #geo 58th and Churchill St.; #geo HP Pavilion Building); a parsing algorithm could determine the format use and a translation tool would convert to lat long. Google seems to use something like this in the mapping tool, no? I suggest '#geo' which is longer than "#ll", but easier to understand if you were picking this up on the fly.

More thoughts:
Standards develop by critical mass in Twitter, and they evolve rapidly. A translation tool could be a way to get around the fact that the masses don't use pre-described standards. A flexible (learning) translation tool could identify emerging crises by watching out for high use of potential tags (e.g.,  #fire, #fireline, #frlne, #lin, #flin etc. could be potential tags used to id a fireline, if lots of tweets start using one, you may have distinguised a growing forest fire from a campfire).

CAP standards seem primarily for dissemination of official information; FEMA may not want Joe Schmoe's to Twitter official sounding reports, because of the subjectiveness of elements such as 'severity'. It would be great, however, for FEMA release CAPs alerts via Twitter (i.e. in 140 characters) - a useful challenge for Twitter.

Keep any more ideas coming; Kate and I will be talking with her group on hashtags who work on Crisis informatics to flesh out the idea.

We've been working with non-official sounding reports... they are already out there. People are generating tons of info with their own tags - but it is very to train algorithms to pick this up - and to adapt to changes within the language. (We haven't decided whether the 140 character limit makes it harder or easier... definitely different though). A quick peak at the history of AI and NLP shows a lot of broken promises. Our lab is currently devoting lots of resources to NLP - and I am sure there are many others out there doing the same.

The value of tweak the tweet is that is doesn't rely on machine-learning and NLP. Emergency responders wouldn't distribute this directly... we could write algorithms to make sense of it and make sure it is this best possible information (as opposed to "accurate" which turns out to be an impossible measure). Once processed, it would extend back out through the more formal channels. Our research indicates that Twitterers still trust their traditional sources (local media, emergency response agencies, etc) the most. So, I would argue that we can allow users to Tweet "unofficial" reports, but then we don't redistribute them through more formal channels until they've been validated as best we can.

On Tue, Dec 1, 2009 at 12:05 PM, Mike Pegg <mp...@google.com> wrote:

Hey there Ryan,

Hope the week back is off to a good start for you.. I wanted to introduce you to a Jeff Martin who is a close colleague of mine here on the Geo team.  He was one of the organizers of the recent Random Hacks of Kindness event and there was some interesting ideas that came out of that event specific to disaster response that involves Twitter and potentially the Twitter API.  I'll let Jeff detail more of the concept by email to see if this is something you might be able to comment on.

Thanks! 

Mike.

 

--
Mike Pegg | Google Inc. | mp...@google.com
Product Marketing, Google Geo APIs
http://maps.google.com/getmaps

--
Jeffery Martin
Business Product Manager  |  Google Crisis Response Team

--
Jeffery Martin
Business Product Manager  |  Google Crisis Response Team

--
Jeffery Martin
Business Product Manager  |  Google Crisis Response Team

--
Jeannie A. Stamberger, PhD
cell: (650) 380-1158

--
Jeannie A. Stamberger, PhD
cell: (650) 380-1158

--
Jeannie A. Stamberger, PhD
cell: (650) 380-1158

--
Jeannie A. Stamberger, PhD
cell: (650) 380-1158

[M. Edward (Ed) Borasky]

1. I am definitely going to Chirp. I signed up yesterday.

2. My understanding is that there are at least 800 seats and they are rolling out in waves. So if someone wants in, keep trying!

3. I sent an email to Ryan Sarver yesterday suggesting that "someone from the CrisisCommons / CrisisCamp effort" give a presentation to the conference about the Twitter-related accomplishments in the aftermath of the Haiti quake. I hope that happens!

4. I don't think this was anyone I know, and it certainly wasn't me, but I can certainly forward the requests to people I've been working with. I've been working with the Twitter API for many months, I'm on the developers' mailing list and can help people project-wide coding to the API.

Let me see what I can dig up.

--
M. Edward (Ed) Borasky
http://borasky-research.net

"I've always regarded nature as the clothing of God." ~Alan Hovhaness

[mark chance]

Hey all -

What is Chirp?  ... link?

Re: @haiti_tweets - fyi - the code I had done Saturday before last was using the @haiti_tweaked account only.  Good luck resolving the mystery.

Cheers,

Mark

[Sam Gilbert]

1. I just registered for Chirp earlier today. I'd love to see a nice contingent of CrisisCommons/ CrisisCamp folks there!  the link: http://chirp.twitter.com

2. Meanwhile, has there been any movement on Tweak the Tweet since the weekend? I and at least of few others from CC Boston are really excited to move forward on Tweet Triage, our sub project. 

3. It'd be nice to coordinate next steps. If others are up for it, I'd be happy to set up a conference call later this week. People free Thursday at 10pm EST?

4. I've been working to gather as many tweets relevant to haiti as possible before they disappear from the search API. I'm sitting on a database of ~3.5 million tweets, with more coming in, and I hope to combine this set with tweets from TwapperKeeper and other sources. If you have other tweets, or if you want my tweets, just let me know!

5. Haven't used the haiti_tweaked account at all, sorry. 

Best, 

Sam

[M. Edward (Ed) Borasky]

OK ... took a second look at the email. A brief course in working with Twitter might be useful - apologies if this is old news:

1. There is a Google Group for developers - people like me who work with the Twitter API, rather than working with Twitter on the web or with someone's desktop or server Twitter tool. That group is

http://groups.google.com/group/twitter-development-talk

I'm on there, and anyone who is working with the Twitter API from any project should be there, if only as a lurker. The most up-to-the-minute documentation can be found there.

2. There appear to be multiple support channels into Twitter for different classes of users. Again, for developers, the one I'd recommend is sending an email to a...@twitter.com. You'll get an email back with an issue tracking number. They're the ones that can deal with blocking, whitelisting, etc.

For lower-frequency requests, there is a Twitter issue tracker at

http://code.google.com/p/twitter-api/issues/list

The Twitter API documentation is at

http://apiwiki.twitter.com/Twitter-API-Documentation

[M. Edward (Ed) Borasky]

Chirp is the Twitter Developers' Conference:

http://plancast.com/a/b3c

[M. Edward (Ed) Borasky]

1. So far, I know I'm going, and some people from the Swift project are organizing a meetup. Plancast is probably the way we want to self-organize CrisisCommons / CrisisCamp activities at Chirp.

http://plancast.com/a/b3c

2. The CrisisFilter project (Crisis...@googlegroups.com) is definitely working with Twitter data, curating tweets and tweeters and creating feeds into other systems. We got a lot of that laid out at CrisisCampPDX Saturday, and at the moment, hacking appears to be happening. My own piece of that has mainly been as a consultant on the Twitter API - it's a Rails project and I'm not a Rails programmer - been working in Perl for nine months and need to get back up to speed on Ruby to be able to do more.

3. I'm up for a conference call, but I'm tied up Thursday, and I'm not sure I'd be able to contribute. I'd guess the thing to do would be to join Crisis...@googlegroups.com and discuss project synchronization there.

4. I suspect you have the largest tweet database by far. Which hashtags are we using in TwapperKeeper? What other sources do you have? I have a few small pieces I've pulled directly from the API for various tests, but I'd be very surprised if I have anything you don't have or TwapperKeeper doesn't have. CrisisFilter is also collecting tweets via Yahoo Query Language.

So - I'm interested in your tweet database but not ready to write any code to deal with them just yet. What's the format? Raw JSON? Raw XML? Some RDBMS? CSV?

[Chris Blow]

Wow 3.5m -- how were those gathered? (which sources, feeds, searches,
and technical methods).

best
c

[Sam Gilbert]

They're all pulled from Twitter's Search API, matching on "haiti"--which means there's a lot of noise in there as well, but I was trying to be good and thorough.  

Instead of using the built-in pagination, if you iterate back through tweets using the max id parameter--that is, if you set the max id as the status id of the final tweet of the previous call--it's possible to circumvent Twitter's pagination limit and get ~6-8 days worth of tweets. 

Best, 

Sam

[M. Edward (Ed) Borasky]

yes, I did that the other day and pulled a dataset going back to 14 January UTC. But the dang Search API only gives you location data if the user geotagged or if you asked search for results inside a geocode circle. Guess the Twitter folks didn't design their API for post-earthquake relief efforts. ;-)

I just searched on the hashtag #haiti, so I have less noise. ;-) It's still 176 MB uncompressed JSON.

So I'm guessing my dataset is redundant. The query for the Twitter Trending Topic was

"haiti OR #haiti"

and that should match what you have. What format is yours? JSON? XML? CSV? Does it need "flattening" (does it have embedded location objects)??

Meanwhile I am nibbling away at getting something built in Ruby for geotagged tweet collection from the Streaming API. "Nibbling" is a better description than "hacking", but I'm hoping to have the PDX collector running some time tomorrow so we can push geotagged tweets and verify they're coming down the pipe. At least those of us in the area bounded by a few miles N of Vancouver, a bit N of Salem, a bit W of Forest Grove and a bit E of Gresham can. ;-) There's only a single bounding box for PDX - Haiti will require nine the last time I checked, although we can trim that down by ignoring a chunk of ocean if we have to.

[Sam Gilbert]

Yeah, the Search API is really frustrating--no geotagging or user data, plus the user ID you get is not in fact a user's REST ID. 

The tweets I pulled were inserted directly into a Postgresql database--happy to export them as a CSV or DB dump if you'd like 'em. 

Best, 

Sam

[M. Edward (Ed) Borasky]

I heart PostgreSQL - a compressed dump would be fine for me. Anybody else want the data?

[Chris Blow]

ed, 

can you put it on github? 

c

[M. Edward (Ed) Borasky]

Depends on how big it is. I have a 600 MB limit of which 300 MB appears to be in use. But I'm guessing I have a fair amount of stuff I can purge. What kind of bandwidth would be hitting it? I might be able to put it up on my web host as a PostgreSQL database or just as an FTP download.

[M. Edward (Ed) Borasky]

Yup - just got rid of 120 MB of useless benchmark log files. I don't remember why I posted them, actually ;-)

[Chris Blow]

please do, that would be great. 

c

[M. Edward (Ed) Borasky]

On Tue, Jan 26, 2010 at 7:27 PM, Sam Gilbert <sam.o....@gmail.com> wrote:

Yeah, the Search API is really frustrating--no geotagging or user data, plus the user ID you get is not in fact a user's REST ID. 

The tweets I pulled were inserted directly into a Postgresql database--happy to export them as a CSV or DB dump if you'd like 'em. 

OK ... go ahead and export them as CSV and compress the file. Let me know how big it is and where it is and I'll deal with it from there. gzip, bzip2 or zip is fine - it should compress about 10:1 with any of them.

[Seth Woodworth]

Sam,

Let's go ahead and put this in /var/www/ on $toast (ping me for
details). It doesn't make sense for Ed to have to put this on github,
we can just make it world accessible.

--Seth

[M. Edward (Ed) Borasky]

Thanks!! Let me know where it is when it's up.

[Sam Gilbert]

Hey all, sorry for the delay--I've finally made the ~3.5 haiti tweets I collected available. You can download a zipped CSV file of them (with a README) here: 

 http://toast.hellosilo.com/haiti_tweets.tar.gz  (it's about 345MB)

Hope they come in handy! I'll continue to collect tweets in this way, so if you'd like a more up to date sample, just email me and I'll export a new csv. 

Best, 

Sam

[M. Edward (Ed) Borasky]

Thanks!! Downloading it now!!