Nameko and multiprocessing

[Manfred Chebli]

Hi folks,

We are currently using nameko in my team, and we are happy with it for the most part.

We would like to launch heavy computation from nameko services, and those computations needs to be done in parallel. I'm not talking about a nameko entrypoint, but an entrypoint that calls a code that needs to be run on several processes.

As of today, we are using python's multiprocessing : https://docs.python.org/2/library/multiprocessing.html.

We tried to wrap our code into a nameko service, and everything went south when using the library, namely because it doesn't play nicely with eventlet.

Has anyone ever used multiprocessing with a nameko service? Or is there a preferred way to run computations in parallel with nameko (with or without multiprocessing)? Of course, we have already tried to use greenthreads for our computations. As they produce a lot of writing operations on a database, performances were not good, even worse than what we are achieving without multiprocessing at all.

Thanks in advance,

Manfred

[Matt Yule-Bennett]

Hi Manfred,

I've not tested it, but I would not expect Eventlet and Multiprocessing to play well together.

You should not need multiprocessing though. Nameko will happily run many concurrent entrypoints. The concurrency is controlled by the `max_workers` config key. The default value is 10 so if you've not changed it, it's probably restricting throughput.

A limitation is that a service's concurrent workers all run within the same Python process (unlike multiprocessing, which spawns a new Python process for each job). Python processes are restricted to a single core, so on a multi-core machine multiprocessing will perform much better than a single Nameko service (if the work is CPU-bound). You can work around this by simply running multiple instances of your service.

Matt.

[Johan Frisell]

One simple way we achieve this is to have nameko dispatch events which are handled by another nameko service or the same service dispatching the event. Might be a naive way to solve the problem but it kinda fits the solution pattern in general and the code base becomes quite neat :)

[Chebli, Manfred]

Thanks you two for your replies.

@Johan I'm not sure I understand your solution correctly. I would like to use multiprocessing _after_ the nameko worker has been spawned. As nameko workers are spawned in process-bound greenthreads, you can use multi-processing in those.

I don't see why using an event entrypoint would change any of this?

Cheers,

Manfred

--
You received this message because you are subscribed to the Google Groups "nameko-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nameko-dev+unsubscribe@googlegroups.com.
To post to this group, send email to namek...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/nameko-dev/847e605a-8cb6-417f-a985-35ff751c228d%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Manfred Chebli     | manfred...@forcity.io

[Matt Yule-Bennett]

Manfred,

Did you come to a solution for this in the end?

Presumably the reason for using multiprocessing was to escape the limitations of the GIL, and use more than one core. I think what Johan was suggesting was to ditch multiprocessing altogether -- if you distribute the work using events, then each one can be picked up by a separate service instance (running in a separate process).

Matt.

On Tuesday, September 5, 2017 at 2:31:49 PM UTC+1, Manfred Chebli wrote:

Thanks you two for your replies.

@Johan I'm not sure I understand your solution correctly. I would like to use multiprocessing _after_ the nameko worker has been spawned. As nameko workers are spawned in process-bound greenthreads, you can use multi-processing in those.

I don't see why using an event entrypoint would change any of this?

Cheers,

Manfred

[juko...@gmail.com]

Running a MultiProcess pool inside a Nameko process is super-hairy (I used concurrent.futures).

You end up with all sorts of ghost services, that you can no longer manage via Supervisord (which is what I use), and you have to kill them manually. I spent a decent amount of time wondering why my workers were returning results from outdated code!

My fix, was to switch to MultiThreading, and use that to marshall a collection of Nameko processes. In essence, I have a top-level Nameko process, which then spawns additional processes. With this, I can distribute the processing load across hosts, as well as processors.

Geoff

[Chebli, Manfred]

Hi Matt,

Sorry for the late reply.

We still have a same solution: we use nameko services as an entrypoint, and then delegate the work to another process, that can span new python processes instead of greenthreads. It's not ideal, and we are looking to improve this part in the future, but we did not took the time to reflect on this at this point.

> Presumably the reason for using multiprocessing was to escape the limitations of the GIL, and use more than one core. I think what Johan was suggesting was to ditch multiprocessing altogether -- if you distribute the work using events, then each one can be picked up by a separate service instance (running in a separate process).

This could be a good solution, but this will need some refactoring here and there in our code before we can implement this. I'm not sure some refactoring can be avoided, so I'll keep this idea in mind.

Thanks for your help on this matter.

Manfred

--

You received this message because you are subscribed to the Google Groups "nameko-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nameko-dev+unsubscribe@googlegroups.com.
To post to this group, send email to namek...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/nameko-dev/a2444bdf-8b14-4d1d-9784-3a2d89523b59%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Manfred Chebli  Ingénieur Logiciel  04 78 18 75 38 | manfred...@forcity.io