How do I set up a pool of workers for my app

[Austin Holland]

So I have developed a super cool app using circuits, but I would like it to distribute the processing load over several threads as I built it to take advantage of my many processors and circuits concurrency capability.  How do I enable a pool of workers within my app?

Thanks,

Austin

[James Mills]

Hi Austin,

Haven't seen you about before? Tell us more about your super cool app? :) I'm intriuged!

To answer your question -- easy:

from circuits import task, Worker

Worker().register(manager)

Later in your code you can do things like:

self.fire(task(f, *args, **kwargs), "worker")

Where:

f is a function

args obviously arguments passed to the function

kwargs obviously keyword arguments passed to the function

For an example of this in use in something I've built see:

https://bitbucket.org/ccaih/ccav/src/tip/src/ccav/api/map.py#cl-135

Hope this helps!

cheers

James

James Mills / prologic

E: prol...@shortcircuit.net.au

W: prologic.shortcircuit.net.au

--
You received this message because you are subscribed to the Google Groups "circuits" group.
To unsubscribe from this group and stop receiving emails from it, send an email to circuits-user...@googlegroups.com.
To post to this group, send email to circuit...@googlegroups.com.
Visit this group at http://groups.google.com/group/circuits-users.
For more options, visit https://groups.google.com/d/optout.

[Austin Holland]

My app is an earthquake processing application that examines data in real time and after the fact.   It uses events to start different processing and reporting as new earthquakes are detected, old earthquake data is modified, new waveform data becomes available, or other types of events occur.  It has a web interface, of course, which can also fire events for the processing and interact with the underlying databases and data.  

I have some heavy calculation signal processing routines, which I could simply send to a worker as you described.  I also discovered the example in the source that uses that as well.  Because I don't know what will be coming when and which method may be doing the most work at any time, I was hoping I could set up a pool of workers and if a worker was idle then the event would get processed by an available worker.  So a somewhat load balanced system.  

Is it possible to open a few to several workers and on the items that usually take some processing time start them as a task.  It straight forward for one worker, but not clear if it works for multiple workers.

Thanks,
Austin

[James Mills]

Your project and use-case for circuits sounds perfect. How would you like to contribute to our Users[1] page?

So what I described and linked to simply wraps around Python's multiprocessing library;

specifically it's Pool class/object. Any time you fire a task() event it gets processed by

an available process in the worker.

So the Worker() component provides you with a worker pool of processes that react to task() events.

I think I'm misunderstanding how you're trying to use it -- but when you initially described what you were

trying to do I think Worker() should do what you want :)

Let me know if you're after something different?

cheers

James

1. https://bitbucket.org/circuits/circuitsframework.com/src/tip/pages/Community?at=default

James Mills / prologic

E: prol...@shortcircuit.net.au

W: prologic.shortcircuit.net.au

[Austin Holland]

Once I'm not embroiled in transitioning stuff to our circuits implementation I would be glad to submit to the Users page as well as perhaps help stub out some documentation.  Although I am already convinced I could do a better job of coding and organization if I was to start over.

The implementation is perfect for what I am looking for.  How do I start multiple workers or is it like multiprocessing Pool where it tries to identify the optimum number from the system?

[James Mills]

On Thu, May 29, 2014 at 1:21 PM, Austin Holland <holland...@gmail.com> wrote:

The implementation is perfect for what I am looking for.  How do I start multiple workers or is it like multiprocessing Pool where it tries to identify the optimum number from the system?

See: http://circuits.readthedocs.org/en/latest/api/circuits.core.workers.html#circuits.core.workers.Worker

Actually the doc strings here are incomplete.

Needless to say you can control the no. of "processes" via the "workers" keyword argument.

Yes it wraps around multiprocessing.Pool()

Also I hope you are using circuits 3.x?