best practices of managing tornado-based services

[elephantum]

Hi!

I'm wondering what's preferred (how does friendfeed) way of
controlling tornado-based service.

In production there are several common tasks to perform with any
service:
- start
- stop
- reload/restart (in case of config change)
- query status (is service running?)

I'm especially interested in "stop" and "status" actions.

At the moment I've implemented this actions as special url-handlers.
I.e. starting of server means starting several processes by the number
of cores on individual ports (8000-8003 for example). Then "stop"
means something like "curl localhost:$port/stop/" for $port in
8000-8003.

I do not really like this approach, but I cannot think of anything
more elegant.

So what is your way of managing tornado-based services?

[Creotiv]

Better way it's to wrap server with daemon. If you want to get status
from them you'll need to use pipes with processes and rewrite tornado
to work with them and send status.

[Bret Taylor]

We use standard Unix methodology to start/stop/restart/etc (PID files, etc). We ping by curl'ing a known URL.

Bret

[Gabriel Farrell]

I don't know what Bret uses, but daemontools
(http://cr.yp.to/daemontools.html) is widely employed for this kind of
thing. A little tricky to set up, but it handles starting, stopping,
logging, etc.

I haven't tried supervisord (http://supervisord.org/), but it looks
interesting. It's written in Python, so it must be good, right?

[Matthew Ferguson]

Supervisord (http://supervisord.org) is great for these situations, along with its own init scripts. You can check out a tutorial on exactly such a thing at http://www.jeremybowers.com/blog/4/tornado-web-framework-production-django-and-nginx/

[Deepankar Sharma]

I am setup a supervisord + tornado machine some time back following
that tutorial and everything works fine and dandy. if you run into any
issues just ping me and I will try to help.

[elephantum]

Bret, does this mean, that you do not use recently introduced
HTTPServer.start(num_processes=N) ?

On Jan 11, 11:29 pm, Bret Taylor <btay...@gmail.com> wrote:
> We use standard Unix methodology to start/stop/restart/etc (PID files, etc).
> We ping by curl'ing a known URL.
>

[elephantum]

Thanks everyone for supervisord suggestion, it seems really nice.

The only thing I don't really like that it introduces another way of
managing processes alongside with /etc/init.d/$service approach.

On Jan 12, 12:02 am, Deepankar Sharma <deepankar.sha...@gmail.com>
wrote:

> I am setup a supervisord + tornado machine some time back following
> that tutorial and everything works fine and dandy. if you run into any
> issues just ping me and I will try to help.
>
>
>
> On Mon, Jan 11, 2010 at 3:58 PM, Matthew Ferguson <mbf...@gmail.com> wrote:

> > Supervisord (http://supervisord.org) is great for these situations, along with its own init scripts.  You can check out a tutorial on exactly such a thing athttp://www.jeremybowers.com/blog/4/tornado-web-framework-production-d...

>
> > On Jan 11, 2010, at 12:55 PM, Gabriel Farrell wrote:
>
> >> I don't know what Bret uses, but daemontools
> >> (http://cr.yp.to/daemontools.html) is widely employed for this kind of
> >> thing. A little tricky to set up, but it handles starting, stopping,
> >> logging, etc.
>
> >> I haven't tried supervisord (http://supervisord.org/), but it looks
> >> interesting. It's written in Python, so it must be good, right?
>

> >> On Mon, Jan 11, 2010 at 3:29 PM, Bret Taylor <btay...@gmail.com> wrote:
> >>> We use standard Unix methodology to start/stop/restart/etc (PID files, etc).
> >>> We ping by curl'ing a known URL.
> >>> Bret
>

[Stephen Day]

I will put in another vote for supervisord. If you want to manage those processes with init, you can put a little script in /etc/init.d that aliases start,stop,restart, etc. to supervisord. 

[mattd]

supervisord is great, but it doesn't (from my usage on osx and ubuntu)
kill all of the forked processes when using the new HTTPServer.start
(num_processes=N) method. im still using 4 separate program
configurations in supervisor and then adding them to a group conf for
batch control.

[Stephen Day]

That's probably a bug in the way tornado is spawning the daemons or a bug in the way supervisord kills the process. Do your forked processes hang if you send a regular sigquit?

In any case, you should go ahead and file a bug on github, if anyone doesn't object.

_steve

[apit]

i use supervisord too :). this is a `ps axwf` when tornado use
forking:

9126 ? Ss 0:00 /usr/bin/python /usr/local/bin/supervisord
9548 ? S 0:01 \_ /usr/bin/python -W
ignore::DeprecationWarning /rinjani/app/main.py --port=9999 --
logging=error
9555 ? S 0:00 \_ /usr/bin/python -W
ignore::DeprecationWarning /rinjani/app/main.py --port=9999 --
logging=error
9556 ? S 0:00 \_ /usr/bin/python -W
ignore::DeprecationWarning /rinjani/app/main.py --port=9999 --
logging=error

after supervisorctl stop:

9126 ? Ss 0:00 /usr/bin/python /usr/local/bin/supervisord
9555 ? S 0:00 /usr/bin/python -W
ignore::DeprecationWarning /rinjani/app/main.py --port=9999 --
logging=error
9556 ? S 0:00 /usr/bin/python -W
ignore::DeprecationWarning /rinjani/app/main.py --port=9999 --
logging=error

i have no idea about process/fork. what happen?

[klokie]

On Jan 28, 1:07 am, Stephen Day <stevv...@gmail.com> wrote:
> That's probably a bug in the way tornado is spawning the daemons or a bug in
> the way supervisord kills the process. Do your forked processes hang if you
> send a regular sigquit?

Yes, that's what I experienced too.

I'm able to successfully stop or restart the forked processes by
adding
"stopsignal=INT" to the command configuration.

cheers
Klokie

> In any case, you should go ahead and file a bug on github, if anyone doesn't
> object.
>
> _steve
>

[Hari]

Does Tornado's pre-fork mechanism make use of a system-wide mutex so
that only one process can do accept/poll at anytime? From what I read,
if you don't use a mutex, then all processes get woken up when there
is an incoming connection, but only one of them would accept and the
rest simply go back to wait, leading to some inefficiency.

The reason I am asking this is to know if the new pre-fork mechanism
is preferable over spawning multiple instances, each on a separate
port, especially if there is going to be something like nginix in
front of them anyway. Is there any consensus on this matter?

Regarding supervisord, multiple instances can be specified more
efficiently in a single group using the "numprocs" property, something
like this (taken from http://supervisord.org/manual/current/configuration.html):

[program:tornado]
numprocs = 4
command = python main.py --port 80%(process_num)02d

The above assumes that the port number to listen to can be specified
using an option. The guide that
Matthew Ferguson linked to earlier seems to create multiple wrapper
scripts one for each port, which seems clumsy to me.

-- Hari

--
Subscription settings: http://groups.google.com/group/python-tornado/subscribe?hl=en