Maximum execution time exceeded (high volume of tweets written to a file)

[Jon]

Hello,
I'm using Phirehose to track keywords and write to a file. It seems to
work fine at low volumes of tweets, but at high volumes has been
getting the following errors:

[21-Feb-2012 16:28:16] PHP Fatal error: Maximum execution time of 30
seconds exceeded in /home/intent/phirehose_live/lib/Phirehose.php on
line 371
[21-Feb-2012 18:36:40] PHP Fatal error: Maximum execution time of 60
seconds exceeded in /home/intent/phirehose_live/lib/Phirehose.php on
line 409

I upped the set_time_limit in the 2nd example to 60 seconds.

Is it a bad idea to use "set_time_limit(0)"/is there a better solution
anyone has found for this problem?

Thanks!
Jon

[Darren Cook]

> I'm using Phirehose to track keywords and write to a file. It seems to
> work fine at low volumes of tweets, but at high volumes has been
> getting the following errors:
>
> [21-Feb-2012 16:28:16] PHP Fatal error: Maximum execution time of 30
> seconds exceeded in /home/intent/phirehose_live/lib/Phirehose.php on
> line 371
> [21-Feb-2012 18:36:40] PHP Fatal error: Maximum execution time of 60
> seconds exceeded in /home/intent/phirehose_live/lib/Phirehose.php on
> line 409

Comparing those timestamps, it used 60 seconds of CPU in 2 hours, which
is 0.08% CPU usage.

I just checked, and one of my scripts has been running 7 days, and so
far has used "208:59" (CPU time, mins:secs). It is using 46MB of memory
apparently, and seems quite constant at 1.9% CPU usage. (IIRC that is
1.9% of one core.) It collected 1.4 million tweets in that time.

So your numbers seem about right.

I'm not explicitly calling set_time_limit() (and phirehose does not
either) but the manual (
http://jp.php.net/manual/en/info.configuration.php#ini.max-execution-time )
says max_execution defaults to 0 from the commandline, and "php -i |grep
execution" does indeed tell me zero.

> Is it a bad idea to use "set_time_limit(0)"

I think not, but if your code has any potential for an infinite loop you
may want to rethink that (*). E.g. one idea would be to call
set_time_limit(20) in your statusUpdate() function (which is called
every minute).

Darren

*: Also remember your phirehose script should be doing as little work as
possible, little more than saving the tweets somewhere, so as not to
hold up the streaming socket.

--
Darren Cook, Software Researcher/Developer

http://dcook.org/work/ (About me and my work)
http://dcook.org/blogs.html (My blogs and articles)

[Darren Cook]

> Comparing those timestamps, it used 60 seconds of CPU in 2 hours, which
> is 0.08% CPU usage.

(typo, sorry!) 0.8%

Darren

[Jon]

Thanks for the quick response!

That makes sense. I doubt I would've figured out that the CPU time was
building up over time and not from a single call. Thanks.