Monitoring scheduler with supervisord

[Yarin]

I'm trying to use supervisord to monitor scheduler workers. 

My supervisord.conf file:

[program:my_scheduler]
command={my_site_path}/web2py.py -K my_app

When I launch supervisord, a worker shows up in the scheduler. However, when I compare the listed pid in the supervisord console, it doesn't match up with the pid of the worker as found in the worker name. Sure enough, when I stop or kill the process in supervisord, it has no effect on the worker, which keeps on ticking. Meanwhile, the 'killed' worker in the supervisord console now shows a pid = 0.

This confuses me because when you launch a worker directly in a terminal window, it doesn't daemonize- so I don't understand why supervisord loses control of it, and why there are 2 different PIDs. Are there subprocesses being launched im not aware of, and if so does anyone have an idea of how supervisord can track them?

[Niphlod]

Because web2py lets you launch multiple schedulers with web2py.py -K myapp,myapp or web2py.py -K myapp1,myapp2 the actual scheduler process is a subprocess of the main one you launch with web2py.py.

If you launch with the "external" method, as python gluon/scheduler.py -t tasks.py -f .... etc no subprocess is spawned and then the PID reported on the scheduler_worker table is the same one you have on "console".

Obviously a scheduler spawns a process for every task processed (that's the whole point of the scheduler) but there's no "master" spawning the scheduler itself.

BTW, I'm working on having the "embedded mode" to terminate all workers processes spawned when the "master" is killed. It's on the feature list.

PS: for the moment you can make a script to work as "embedded mode" and with no subprocesses spawned. Place it in the same folder where web2py.py lives. This will run a single scheduler process with no subprocesses

#!/usr/bin/env python
# -*- coding: utf-8 -*-
import os
import sys

if '__file__' in globals():
    path = os.path.dirname(os.path.abspath(__file__))
    os.chdir(path)
else:
    path = os.getcwd()

sys.path = [path]+[p for p in sys.path if not p==path]

from gluon.shell import run
from gluon import main
import logging

logging.getLogger().setLevel(logging.INFO)

if __name__ == '__main__':
    code = "from gluon import current; current._scheduler.loop()"
    app = 'myapp'
    run(app,True,True,None,False,code)

[Yarin]

The embed-mode script works great and allows us to use Supervisord- Thanks--

On Wednesday, August 15, 2012 11:44:18 AM UTC-4, Niphlod wrote:

Because web2py lets you launch multiple schedulers with web2py.py -K myapp,myapp or web2py.py -K myapp1,myapp2 the actual scheduler process is a subprocess of the main one you launch with web2py.py.

If you launch with the "external" method, as python gluon/scheduler.py -t tasks.py -f .... etc no subprocess is spawned and then the PID reported on the scheduler_worker table is the same one you have on "console".

Obviously a scheduler spawns a process for every task processed (that's the whole point of the scheduler) but there's no "master" spawning the scheduler itself.

BTW, I'm working on having the "embedded mode" to terminate all workers processes spawned when the "master" is killed. It's on the feature list.

PS: for the moment you can make a script to work as "embedded mode" and with no subprocesses spawned. Place it in the same folder where web2py.py lives. This will run a single scheduler process with no subprocesses

#!/usr/bin/env python
# -*- coding: utf-8 -*-

from gluon.shell import run
from gluon import main
import logging

logging.getLogger().setLevel(logging.INFO)

if __name__ == '__main__':
    code = "from gluon import current; current._scheduler.loop()"
    app = 'myapp'
    run(app,True,True,None,False,code)

[Yarin]

@Niphlod- I was hoping this "embedded mode" patch was going to become part of the 2.0 implementation. I see you added a -X flag to allow for launching the scheduler alongside an app, but we can't use this in our situation where web2py is running as wsgi under Apache while our scheduler processes run under Supervisord. Thoughts?

On Wednesday, August 15, 2012 11:44:18 AM UTC-4, Niphlod wrote:

Because web2py lets you launch multiple schedulers with web2py.py -K myapp,myapp or web2py.py -K myapp1,myapp2 the actual scheduler process is a subprocess of the main one you launch with web2py.py.

If you launch with the "external" method, as python gluon/scheduler.py -t tasks.py -f .... etc no subprocess is spawned and then the PID reported on the scheduler_worker table is the same one you have on "console".

Obviously a scheduler spawns a process for every task processed (that's the whole point of the scheduler) but there's no "master" spawning the scheduler itself.

BTW, I'm working on having the "embedded mode" to terminate all workers processes spawned when the "master" is killed. It's on the feature list.

PS: for the moment you can make a script to work as "embedded mode" and with no subprocesses spawned. Place it in the same folder where web2py.py lives. This will run a single scheduler process with no subprocesses

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import os
import sys

if '__file__' in globals():
    path = os.path.dirname(os.path.abspath(__file__))
    os.chdir(path)
else:
    path = os.getcwd()

sys.path = [path]+[p for p in sys.path if not p==path]

from gluon.shell import run
from gluon import main
import logging

logging.getLogger().setLevel(logging.INFO)

if __name__ == '__main__':
    code = "from gluon import current; current._scheduler.loop()"
    app = 'myapp'
    run(app,True,True,None,False,code)

[Niphlod]

Scheduler is not useful if managed by a webserver, so forget about -X: just like softcron, it is meant to be used when the internal webserver is used, not when web2py is managed by apache & co.

Having a "master" starting the actual workers is needed to have multiple workers started with one command through web2py.
If you want to manage the workers through supervisord with the standard config you have to use this script.

[Yarin]

Right, I was just voting for having the script integrated into web2py outright, so that calling it with maybe a lowercase "-k" would execute a single scheduler process without spawning subprocesses.

[Niphlod]

well, maybe we can support not spawning if -K is followed by one app only. Thoughts ?

[Yarin]

Sure that works for me

[Niphlod]

ok, give me a few hours, I need to get home first :P

[Yarin]

Any luck with this Simone?

[Niphlod]

no free time at home to test on every platform, but the code part is done. in a few days.

[Yarin Kessler]

You the man

--