SiteStreams OAuth support

[Sebastian Hoitz]

Hi!

I posted my addition to the Phirehose SiteStreams for OAuth support a
while ago. The problem was: It had dependencies on Zend Oauth (Booo!).
The reason was, that the UserStreams OAuth implementation did not work
for SiteStreams. Also this was the quick fix for me in one of my
projects, since I was using Zend anyway.

However, it is a lot cleaner without the Zend dependency.

So since we are officially now on Github I want to bring my first
contribution to this project:

OAuth support for SiteStreams without Zend dependency!

My commit: https://github.com/sebastianhoitz/phirehose/commit/4c1a832d8a725a4ba4...
And the pull request: https://github.com/fennb/phirehose/pull/1

This is mostly based on the awesome UserStreams addition by Jonathan
Hill.

Also, I do have further plans on restructuring the Phirehose class,
since it is not really well suited for extending / changing
authorization right now (see all the duplicate code in there?). Anyone
wanna participate?

Thanks,
Sebastian

PS: Is someone going to TheNextWeb conference or in Amsterdam next
week? Wanna grab some beers and talk about Twitter API, PHP
development and other stuff? Let me know! :)

[Sebastian Hoitz]

I just pushed some more changes into my github repo. I improved how the Authorization headers are generated. This makes the OAuth addition a lot more cleaner in my opinion.

What do you think?

2011/4/24 Sebastian Hoitz <nekro...@gmail.com>

--
Mit freundlichen Grüßen,
Sebastian Hoitz

[Jonathon Hill]

Great work, Sebastian! Glad to see someone has extended Phirehose to support Site Streams independently.

Fenn, it would really be nice if you would bundle versions of the main Phirehose library for user streams and site streams. Then you would have complete coverage of Twitter's streaming API collection!

Jonathon Hill

[Fenn Bailey]

Hey Sebastian/Jonathon,

Great work indeed Sebastian, thanks heaps for that!

Jonathon - You're absolutely right, we definitely should bundle it all up in the distribution so you can access all parts of the streaming API from a single library (something that's not done in any language so far, I don't think).

Sebastian - you're right in terms of how the original Phirehose is not really geared to extending with different authentication/authorization mechanisms, it would make sense to refactor some of it to make it a lot more sane.

I don't have a huge amount of time to do this right now, but it's something that (as a team) we should definitely do. Now that we're on github, it should be a lot easier to integrate code changes from multiple people, or at least, it will be as soon as I learn how to use it :)

On that note, I've added you as a collaborator Sebastian - Jonathon, if you send me your github username offlist, I'll add you as well - I'm sure between us we can get this all cooking along nicely.

We should probably start a refactor branch of some sort so that master doesn't become horribly unstable.

Would love to meet up in Amsterdam for a beer, however, it's a bit of a hike from Melbourne :)

Cheers guys,

  Fenn.

[Adam Green]

While you guys are improving the code I have an issue for you to look at. When Phirehose connects to the streaming API with a very low frequency set of track words, it always dies silently after a few hours of no tweets being received. This is completely reproducible. Just set up a connection with a few track words that nobody would use, and you will eventually see that the connection dies, but the script keeps running and no errors are reported. The only solution is to kill the script and restart.

I have brought this up with Fenn in the past, and the response was that I should debug the code. I understand that this is needed, but this type of code is not my specialty. I'm an application programmer, not a systems guy. This is a serious weakness in the code, and it should be addressed eventually. Right now I deal with it by have a script that kills and then restarts the code running Phirehose every 60 minutes, but that is not a great way to handle this.

--
Adam Green
Twitter API Consultant and Analyst
http://140dev.com, @140dev
http://2012twit.com, @2012twit
781-879-2960

[Fenn Bailey]

Hey Adam (& all),

This is indeed an interesting one - Phirehose already checks for dead streams: https://github.com/fennb/phirehose/blob/master/lib/Phirehose.php#L285

The only way it can "hang" with no tweets is if the streaming API itself is sending keep-alives (blank lines) but no tweets.

You can check this yourself by looking at the logs and looking at the "max stream idle period" number. If this number is low (ie: < 90 seconds), then I'm fairly certain the problem is on Twitter's end (not Phirehose itself) - though happy to be proven wrong :)

What is the idle period you are observing, Adam?

  Fenn.

[Adam Green]

The idle time is being reported as 30 seconds. I'm not sure what the definition of a problem on Twitter's end is. Here is the completely reproducible fact. If you start a Phirehose process that is tracking a tag that nobody uses, it will absolutely fail without an error within a few hours and leave the process running. At that point the script will stop capturing any additional tweets. Whoever is responsible, it happens every time when using Phirehose. So by definition it is a problem that expresses itself as failure when building a Twitter tracking system with Phirehose. I am unable to debug this any further. I don't have that skill set. But a couple of hundred people a month are downloading my 140dev framework based on Phirehose, so I am making my contribution in other ways. It would be great if someone tried to solve this problem.

[Fenn Bailey]

Hey Adam,

I'm checking this out now (it'll take me a couple of hours waiting for the stream to "die"). It should be easy to prove one way or another as I can see the keep-alive packets being sent from Twitter (meaning the stream is "alive") and can just tweet my obscure phrase myself and see if it comes up in the stream.

Will let you know how I go.

Cheers!

  Fenn.