Graceful Shutdown of Workers
bcarey
Are there any recommended ways of gracefully stopping a worker as
opposed to just killing the process? For instance when updating
worker code or for any general maintenance requiring a worker to be
stopped. One option I was considering was to register a shutdown
function within the worker that a job could be sent to, however I was
wondering if there is an existing way of doing this, perhaps by
connecting directly to gearmand and stopping the worker?
Thanks in advance for any help you can provide.
Eric Day
Good question. Right now there is not any built-in way of doing
it. If we had the ability to close the connection via gearmand, the
worker will just keep trying to reconnect. The methods now would be
to either register a shutdown function like you mention or to catch
signals and check/break out of the worker->work() loop when you see
a signal come in. It would be possible to add something into the
protocol, but I'd like to avoid that.
-Eric
Brian Aker
How about we hook up a shutdown into the signal handler?
Cheers,
-Brian
--
_______________________________________________________
Brian "Krow" Aker, brian at tangent.org
Seattle, Washington
http://krow.net/ <-- Me
http://tangent.org/ <-- Software
_______________________________________________________
You can't grep a dead tree.
Eric Day
One issue is you don't always want to takeover the signal handlers in
case the parent application is taking them already (ie, PHP interpreter
process or some custom C worker code). I guess we could have a function
the client can call to handle this if they want, I'd be afraid to do
it by default.
Hmm, perhaps having a shutdown command from server is the best way to
deal with it and avoid signals with embedded/interpreted workers. Of
course that means having to telnet to a gearmand the worker is
connected to.
-Eric
Brian Aker
On May 13, 2009, at 1:59 PM, Eric Day wrote:
> Hmm, perhaps having a shutdown command from server is the best way to
> deal with it and avoid signals with embedded/interpreted workers. Of
> course that means having to telnet to a gearmand the worker is
> connected to.
My thoughts:
1) Build a proper shutdown function.
2) Extend the protocol.
3) Run the function if the signal handler is called.
The above would be for the server. We could dummy up one for a worker
but I think it would be less useful...
Cheers,
-Brian
Joe Stump
On May 13, 2009, at 2:03 PM, Brian Aker wrote:
> 1) Build a proper shutdown function.
The Perl, PEAR/PHP, and Python clients all have hooks for stopping
workers. Python has a stop_if(), Perl has a stop_if call back and the
PEAR client has $monitor callback you can pass to beginWork. Every one
of these are perfectly capable of being used to gracefully shut down
workers.
> 2) Extend the protocol.
I vote no on this.
> 3) Run the function if the signal handler is called.
Exactly. Have your callback for the worker code trap signals and
gracefully kill each worker.
Some notes about how we run/use workers at Digg:
1. We run PHP workers via the daemon utility (http://libslack.org/daemon/
)
2. We gracefully exit the worker every 10 - 20 iterations due to PHP's
horrible memory management. The daemon utility re-HUP's the process
from there. We do this via the $monitor callback.
--Joe
bcarey
Thank you for all of the helpful responses! I have gone ahead and
registered a shutdown function within each worker that when called,
either stops the worker if nothing is currently being processed or
sets a flag that is checked whenever a job completes and then stops
the worker at that time.
On May 13, 5:27 pm, Joe Stump <j...@digg.com> wrote:
> Some notes about how we run/use workers at Digg:
> 1. We run PHP workers via the daemon utility (http://libslack.org/daemon/
> )
> 2. We gracefully exit the worker every 10 - 20 iterations due to PHP's
> horrible memory management. The daemon utility re-HUP's the process
> from there. We do this via the $monitor callback.
>
> --Joe
utility as this also sounds helpful.
- Brian
Ernesto Rodriguez Reina
--
Ernesto Rodriguez Reina
bcarey
gearman and am learning new things about it every day, but here is
what I did. That being said if anyone has any suggestions please
voice them. Keep in mind that the shutdown function would need to be
registered as a specific name for each different worker.
Within the worker
----------------------------------
$gmmanager = new gearman_manager();
$gmworker->add_function( 'shutdown_worker', 'shutdown_worker' );
while (1)
{
$return = $gmworker->work();
....
$gmmanager->stop_processing();
}
function shutdown_worker( $job )
{
global $gmmanager;
$gmmanager->shutdown_worker();
}
Within worker function that is actually called to do work
------------------------------------
$gmmanager->start_processing();
Class definition
------------------------------------
<?php
// Class to manage a gearman worker process.
class gearman_manager
{
private $currently_processing;
private $shutdown_pending;
function __construct()
{
$this->currently_processing = false;
$this->shutdown_pending = false;
}
public function start_processing()
{
$this->currently_processing = true;
}
public function stop_processing()
{
$this->currently_processing = false;
if ( $this->shutdown_pending == true )
{
// Shutdown the worker
$pid = posix_getpid();
echo "Shutting down worker ( pid = $pid )";
exit();
}
}
public function shutdown_worker()
{
$this->shutdown_pending = true;
}
}
?>