Multiprocess cherrypy?

[Dan Stromberg]

We're wanting to be able to reload subregions of our REST (RPC, really) hierarchy without impacting others, and reload() isn't looking like it's necessarily the best way to do it.

Is it possible to have a master process and (EG) 3 subprocesses for /a, /b and /c, all served up by one cherrypy (or 4 cherrypy's)?

Perhaps the parent process on 8080 could just dispatch to 3 other cherrypy's on localhost:8081, localhost:8082 and localhost:8083, all over http?

What would be a good technology for localhost:808{1,2,3} for sending -out- http requests?  Perhaps requests or urllib2?  Or does cherrypy do outgoing too?

Will using an alternative cherrypy disaptcher complicate this?

Thanks!

[Dan Stromberg]

On Tue, Aug 21, 2012 at 11:31 PM, Dan Stromberg <drsa...@gmail.com> wrote:

We're wanting to be able to reload subregions of our REST (RPC, really) hierarchy without impacting others, and reload() isn't looking like it's necessarily the best way to do it.

I've created a stackoverflow question about this:

http://stackoverflow.com/questions/12078974/cherrypy-with-multiprocessing

[Lakin Wecker]

Off the top of my head - one way to do this would be to use NGINX to
proxy to three (or more) internal CP processes. Say,
http://localhost:8080/ and http://localhost:8081/ and
http://localhost:8082 where the process listening on port 8080 serves
up /a and the process listening on 8081 serves up /b, and 8082 serves
up /c.

Then setup nginx to have a custom 502/503 html page - so that if any
one of these processes is down it serves up a nice page explaining the
situation. Then you can restart any of the processes independently
from the rest. You're probably going to want to have nginx proxying
to CP anyways as that seems to be what many of us do these days
anyways.

Honestly - there doesn't even have to be anything different between
the three processes aside from the port they live on. They could be
run the same python/cherrypy configuration/setup and be able to handle
/a/ or /b/ or /c/ - but because nginx never passes anything but one of
the set to that process, then that process will only deal with the one
set of urls.

I don't have time to give you a full explanation of how to setup the
NginxProxy settings, but i'll look something like this:
http://pastebin.com/UVZm6Bsu

You would also likely want to turn on tools.proxy within CP which will
take that Host header and use it, instead of the internal host header
which will always be something like: 127.0.0.1:8080

There are probably other ways, but it was the first i thought of

Lakin

> --
> You received this message because you are subscribed to the Google Groups "cherrypy-users" group.
> To post to this group, send email to cherryp...@googlegroups.com.
> To unsubscribe from this group, send email to cherrypy-user...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/cherrypy-users?hl=en.

Off the top of my head - one way to do this would be to use NGINX to
proxy to three (or more) internal CP processes. Say,
http://localhost:8080/ and http://localhost:8081/ and
http://localhost:8082 where the process listening on port 8080 serves
up /a and the process listening on 8081 serves up /b, and 8082 serves
up /c.

Then setup nginx to have a custom 502/503 html page - so that if any
one of these processes is down



location / {
proxy_pass http://backend_servers;
}

location /files/ {
proxy_pass http://fileserver;
proxy_hide_header Content-Type;

> --
> You received this message because you are subscribed to the Google Groups "cherrypy-users" group.
> To post to this group, send email to cherryp...@googlegroups.com.
> To unsubscribe from this group, send email to cherrypy-user...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/cherrypy-users?hl=en.

[Eric Larson]

For whatever reason, the other day this idea struck me as something that
would be easy to do to. I took a stab at an idea for using ZeroMQ to
send messages on the wspbus. You would have a master cherrypy app that
would be copied and served on different hosts.

To make a long story short, it didn't work as I had hoped. The problem I
ran into was that when I forked the process, cherrypy didn't want to
accept the new cherrypy.config changes.

https://gist.github.com/3433405

Maybe others have thoughts on how to make this work correctly.

In a related note, I released AutoRebuild
(https://bitbucket.org/elarson/autorebuild) which is a simple plugin for
watching files and calling some "build" function. It also has a
ServiceStarter plugin for starting other processes.

You could very easily have a master cherrypy process that only starts
others using the service starter plugin. For example:

import cherrypy
from autorebuild import ServiceStarter

instances = 3
services = []

for instance in range(instances):
service = ServiceStarter(cherrypy.engine)
service.subscribe()
service.cmd = 'python myapp/run.py --port %s' % 9000 + instance
services.append(service)

cherrypy.engine.start()
cherrypy.engine.block()

That should start an instance of some cp app on port 9000, 9001, and
9002.

Let me know if others have comments on the AutoRebuild plugins or my
cpcluster idea.

Eric