Assume you have a server process running, a pool of worker threads to perform tasks, and a Queue.Queue() to pass the tasks to the workers.
In order to shut down the server cleanly, you want to ensure that the workers have all finished their tasks. I like the technique of putting a None onto the queue, and have each worker check for None, put None back onto the queue, and terminate itself.
The main program would look something like this -
q.put(None) for worker in worker_threads: worker.join()
At this point you can be sure that each thread has completed its tasks and terminated itself.
However, the queue is not empty - it still has the final None in it.
Is it advisable to finalise the cleanup like this? -
while not q.empty(): q.get() q.task_done() q.join()
> Assume you have a server process running, a pool of worker threads to > perform tasks, and a Queue.Queue() to pass the tasks to the workers.
> In order to shut down the server cleanly, you want to ensure that the > workers have all finished their tasks. I like the technique of putting a > None onto the queue, and have each worker check for None, put None back > onto the queue, and terminate itself.
> The main program would look something like this -
> q.put(None) > for worker in worker_threads: > worker.join()
> At this point you can be sure that each thread has completed its tasks > and terminated itself.
> However, the queue is not empty - it still has the final None in it.
Yes - but who cares? :)
> Is it advisable to finalise the cleanup like this? -
> while not q.empty(): > q.get() > q.task_done() > q.join()
> Or is this completely redundant?
I don't see what you gain by doing that.
On the other hand, you might check whether the queue contains exactly one item and it is None, just to be sure that all queued work has been done.
(BTW, I'd use a different sentinel instead of None; perhaps this could not happen in your current code, but it's easy to inadvertedly put a None in the queue, stopping all workers prematurely.)
On Feb 6, 7:59 am, "Gabriel Genellina" <gagsl-...@yahoo.com.ar> wrote:
> En Fri, 05 Feb 2010 09:45:30 -0300, Frank Millman <fr...@chagford.com> > escribi :
[...] > > However, the queue is not empty - it still has the final None in it.
> Yes - but who cares? :)
That was my point. I didn't think I needed to care, but I wanted to be sure I was not missing something.
I will just discard all the final cleanup code.
> (BTW, I'd use a different sentinel instead of None; perhaps this could not > happen in your current code, but it's easy to inadvertedly put a None in > the queue, stopping all workers prematurely.)
> Assume you have a server process running, a pool of worker threads to > perform tasks, and aQueue.Queue() to pass the tasks to the workers.
> In order to shut down the server cleanly, you want to ensure that the > workers have all finished their tasks. I like the technique of putting a > None onto thequeue, and have each worker check for None, put None back onto > thequeue, and terminate itself.
> The main program would look something like this -
> q.put(None) > for worker in worker_threads: > worker.join()
> At this point you can be sure that each thread has completed its tasks and > terminated itself.
> However, thequeueis not empty - it still has the final None in it.
> Is it advisable to finalise the cleanup like this? -
> while not q.empty(): > q.get() > q.task_done() > q.join()
> Or is this completely redundant?
> Thanks
> Frank Millman
Queue objects have support for this signaling baked in with q.task_done and q.join.
After the server process has put all tasks into the queue, it can join the queue itself, not the worker threads.
q.join()
This will block until all tasks have been gotten AND completed. The worker threads would simply do this: task_data = q.get() do_task(task_data) q.task_done()
Using pairs of get and task_done you no longer need to send a signal. Just exit the server process and the worker threads will die (assuming of course, you set .setDaemon(True) before starting each worker thread).
> Queue objects have support for this signaling baked in with > q.task_done and q.join.
> After the server process has put all tasks into the queue, it can join > the queue itself, not the worker threads.
> q.join()
> This will block until all tasks have been gotten AND completed. The > worker threads would simply do this: > task_data = q.get() > do_task(task_data) > q.task_done()
> Using pairs of get and task_done you no longer need to send a signal. > Just exit the server process and the worker threads will die (assuming > of course, you set .setDaemon(True) before starting each worker > thread).
Thanks, Steven.
This works perfectly in my scenario, and tidies up the code a bit.
Minor point - according to the 2.6 docs, .setDaemon(True) is the old API - the current way of specifying this is .daemon = True.
