Multiprocess support

109 views

Skip to first unread message

Jeff Lindsay

unread,

Feb 27, 2012, 2:59:45 PM2/27/12

to gser...@googlegroups.com

One big feature we've wanted was being able to run different services
(or more of the same service) in different processes. Multiprocessing
sort of works, but it's not really baked in consistently. And
alternatively there is Pistil, a library by the Gunicorn author that
provides a multiprocess manager made for workers processes that's
really nice.

I think with AsyncManager, we have a place to put multiprocess
support, however, it's not enough. We need to actually be a supervisor
to those long running processes, like Pistol/Gunicorn. Looking at
Pistol code, it's fairly simple, but also makes a lot of assumptions.
There's also a bit of stuff specific to Gunicorn that wasn't removed.
But the biggest thing is that it almost has the service model. There
is a sort of client-server relationship between the child process and
the supervisor/arbiter.

So now! So while any service could have a ProcessAsyncManager (or
however we instantiate different AsyncManagers) for basic
multiprocess-like behavior (simply a pool of processes that run some
function in them), we'd also have a MultiprocessService.

MultiprocessService would inherit from BasicService or maybe
Container. It would force ProcessAsyncManager (but would let you
change it to threads for running tests) and work like this:

- When you call add_service, it will queue it up to be turned into a
child process.
- When you call start, it will take child services and start *them*
in *different* processes using the ProcessAsyncManager.
- It would also have a child service that acts as the supervisor that
would not run in a new process.

In a previous email, I described how all gservice processes would run
under a Process container service. In the case of these new forked
processes, we would take the service you added to MultiprocessService,
and wrap it in a Subprocess service. Subprocess is simply a subclass
of Process that has the "client" that communicates to the supervisor.

MultiprocessService could either be used by itself to create a pool of
workers running the same process:

MultiprocessService(lambda: MyService(), workers=5)

Or you could inherit from it, maybe making it your top-level app service:

class MyApp(MultiprocessService):
workers = Setting('workers')

def __init__(self):
self.add_child(MultiprocessService(lambda: SomeWorkerService(),
workers=self.workers)
self.add_child(SomeOtherService())

If you ran MyApp, it would not only create SomeOtherService() in a
subprocess and make sure it's running. It would create a subprocess
that then manages subprocesses of SomeWorkerService. In other words,
you can create process hierarchies from anywhere, not just a top level
special case of pre-forking.