Phirehose Connection Management

[Gene]

Hey everyone,

I have a question in regards to connection management with Phirehose.
I know that when running methods like checkFilterPredicates, it will
handle the connect/reconnect, etc.What about the code that keeps
Phirehose running? If I am running it as a daemon, does that mean I
need to create an additional system that doesn't restart the script
too soon if it stops running, or can I run the script immediately and
phirehose will know not to reconnect immediately? My main concern is
the "exponential back-ing off" that Twitter says we need to implement.
Thanks for your help.

[Fenn Bailey]

Hey Gene,

Phirehose handles the "exponential back-ing off" internally if the connection drops, however, it (obviously) can't if the script dies.

Ideally, you should try and stop it from dying if you can (ie: catch exceptions and trigger a disconnect() if need be), but presuming it could die, it would be wise to have some sort of "sleep" period regardless, simply because you risk hammering your server if you constantly try and restart in a loop.

There's a bunch of ways to keep scripts running, one of these may suit your needs:

• http://supervisord.org/
• http://upstart.ubuntu.com/
• http://mmonit.com/monit/
• http://cr.yp.to/daemontools.html

Or you could just write a simple bash script. Good luck!

  Fenn.

[Darren Cook]

> handle the connect/reconnect, etc.What about the code that keeps
> Phirehose running? If I am running it as a daemon, does that mean I
> need to create an additional system that doesn't restart the script
> too soon if it stops running, or can I run the script immediately and
> phirehose will know not to reconnect immediately?

As Fenn said it won't know not to reconnect immediately, but it should
not matter, as this is hopefully going to be a rare case.

I run an every-minute cron job, which checks for a keep-alive file every
20 seconds. If the keep-alive file is older than 66 seconds, then it
does a forced restart.
My keep-alive file is created/updated when the script starts (very
important), and then updated once a minute. (In Phirehose terms, I use
the status.log as the keep-alive file, and update it in the
statusUpdate() callback.)

So, my minimum time between forcefully restarting the script is once
every 66 seconds. Two of those back to back is going to be very rare,
and if it happens it would almost certainly mean a network problems
somewhere (implying Twitter would never know about my repeated attempts
to connect).

My maximum time between some failure and trying the forced restart is 86
seconds. My application can cope with that much downtime, but if yours
cannot then you may need a more complicated solution.

Darren

--
Darren Cook, Software Researcher/Developer

http://dcook.org/work/ (About me and my work)
http://dcook.org/blogs.html (My blogs and articles)

[Gene Ellis]

Awesome...that makes perfect sense. Thank you. Once more question. I am speaking about the checkFilterPredicates() function. Everything works fine if I use: $this->setFollow(array($randWord1, $randWord2)); Where  $randWord1 and  $randWord2 are actually user Ids. Like I said, that works fine.

However, I need to follow about 900 people. I have used the global keyword to get my array into the function. However, no matter what I do, it doesn't seem to be following my users. Let's call the array:

$testarray

How would I get the values of $testarray to go into the setFollow method? A total basic question, but I must be missing something. Is there a max input to setFollow? Would it take a really long time to add that many people? I am wondering if I am expecting them to be added to fast.

Thank you very much.

[Fenn Bailey]

Hi Gene,

The follow method takes an array of twitter user IDs - note that these are INTEGER IDs, not twitter usernames, see the twitter documentation for examples:

https://dev.twitter.com/docs/streaming-api/methods#follow

There are limits with the follow parameter, but it's not specified what they are:

https://dev.twitter.com/docs/streaming-api/concepts#filter-limiting

To be honest, 900 people are unlikely to hit your limit, but you can monitor for if you're being limited by looking at the stream:

https://dev.twitter.com/docs/streaming-api/concepts#JSON-2

Cheers,

  Fenn.

[Gene Ellis]

Can you show an example of how I would use $testarray and pass it to setFollow? It seems like something so basic but it won't quit work. If I list out the values like  $this->setFollow(array($randWord1, $randWord2))  it works fine. But I can't seem to get it to read an array. Can you show some sample code of how I would do this? So close to getting it all implemented.

Thank you so much for your assistance.

Regards,

Gene Ellis

[Gene Ellis]

The weird thing is that when I do

$this->setFollow($theArray1) and the array only has a few element, it runs fine...but when I do $this->setFollow($theArray2); and the array has hundreds, nothing happens...The script is running, but nothing is being read on the stream (I have a simple print statement in enqueueStatus). Any idea what could be going wrong. I checked the array and everything. It's such a simple error that I'm wondering if I am missing something stupid. Driving me crazy. Thanks for any assistance. 

[Darren Cook]

> $this->setFollow($theArray1) and the array only has a few element, it runs
> fine...but when I do $this->setFollow($theArray2); and the array has
> hundreds, nothing happens...

The twitter docs mention following 50,000 people, so I don't think
you're hitting an API limit.
I skimmed the source and it is using POST for the data, so I don't think
you'd be hitting a limit there either.

I've had no content returned due to a bad character in my parameters.
(Though that may have been before an RFC encoding bug fix.)

The code is (phirehose.php, line 530):
$requestParams['follow'] = implode(',', $this->followIds);

So it trusts you have good data in your array. One idea is to convert
the array to integer, and see if you ended up with any zeroes or
negative numbers.

This line (in Phirehose.php) says what the request is that it has made:
$this->log('Connecting to twitter stream: ' . $url . ' with params: '
. str_replace("\n", '',var_export($requestParams, TRUE)));

Does it output anything unusual?

[Gene Ellis]

Thanks for the help Darren. I was away for a few days but will give this a try tomorrow and let you know about my findings. Thank you again. 

[Gene Ellis]

Still no dice...I will send you my array. If you don't mind taking a quick glance at it and letting me know if you see anything weird with it, that would be great. I am making sure there are no negative numbers, empty elements, etc..I just don't get why it won't accept my large array.