A New Cron (and post-connection tasks)

[achipa]

The long discussed (and maybe) awaited cron patch is here. Here's a
transcript which should make it more cloear how works ands what's it
good for. If you have any questions, or if you notice something I
messed up, please let me know.

======
This does a couple of interesting things. It can dispatch things to be
executed after the client is
gone, and in a programmable way (it's in a not pretty yet, but in a
works-as-intended phase). You can put this into a controller

class dummy(gluon.contrib.wsgihooks.PostConnectionTask):
def run(self):
logging.warning("hello")
from time import sleep
sleep(10)
logging.warning(self.env)
logging.warning("world")

def mycontrollerfunc():
gluon.contrib.wsgihooks.localposttasks.append(dummy(request))

Localposttasks are those that get executed once and are then
discarded,
globalposttasks are those that should be executed after each
connection.
Maybe it would make sense to wrap this nicely into a T2 feature.
Someone with more time and python skills could make this work as
decorators, which should be a bit cleaner.

========

Cron patch. Simplified it a bit. The postconnection part works as did
previously, the cron works more like a regular crontab. It looks for
an
applications/[application]/cron/crontab file which is in regular
crontab
format. Three modes of operation are available:

- softcron: cron routines are checked after page content has been
served, does
not guarantee execution precision. For unprivileged apache CGI/WSGI
installs.

- hardcron: a cron thread gets started on web2py startup. For windows
and
cherrypy/standalone installs.

- extcron: cron functions get force-called from the command line,
usually from
the system crontab. For unix systems and places where the cron
triggers need
to be executed even if web2py is not running at the moment. Also good
for
CGI/WSGI installs if we have access to the system crontab.

======

> 1) you do you modify wsgihandler? Do I get the cron only if I use
> this handler?

Yes (it is modified) and no (you get it also with cherrypy).
wsgihandler gives
you softcron by default. Either way (cherrypy or wsgi), can opt for
extcron
if you have system crontab access (nice, but not required). I might
have
forgotten to patch the fastcgi handler, will check that (I don't have
a
fastcgi setup at hand).

> 2) your cron jobs are executed in threads. Will the web server kill
> them if they exceed max time?

Hardcron (cherrypy handler) and extcron no, WSGI/fastcgi softcron also
no
(with the exception of having the webserver or the wsgi/fastcgi task
restarted itself). The only execution combination that is limited by
webserver max time should be plain CGI. We *could* implement a
maxtime
checking of our own to avoid zombie threads accumulating for bad cron
tasks
(you don't have that with regular cron either, so not sure about
that).

> 3) If we have multiple web2py installations, only one process should
> run the hardcron. Am I wrong? How is this handled?

All types of cron check for admin/cron/cron.master dates. Only the
instance
that manages to modify cron.master will execute the cron check cycle.
However, your question made made me wonder that you might want a
'disable
cron' parameter in web2py to make sure all cron tasks are started by
the same
instance.

> 4) the soft crons, in which environment are they executed, do they
> see request, response, session, etc?

No, they are executed completely outside web2py by the wsgi callback
function
(=they are executed in the wsgi/cgi environment, but without the
ability to
return a response to user as the connection has already been closed).

> 5) It is important that no reference to SQLDB objects is passed to
> the cron threads or this may break web2py. It looks like you are fine
> but can you confirm?

Confirmed. Cron threads by default do not use anything from gluon. For
cases
where you DO need the DAL or gluon functionality, the cron actually
creates/invokes a new web2py environment and runs the controller/task
there
(so it gets it's own connection, objects, etc).

[mdipierro]

Thank you Achipa,

I will take a look at this tonight.

Massimo

[achipa]

OK, I just realized I can't attach files to postings. So, until I
figure out what am I not seeing in the google groups interface, the
patch can be downloaded from http://www.atombiztos.hu/cron-2009-01-08.patch

[mdipierro]

please send me an email so that it does not get lost.

Massimo

On Jan 8, 10:49 am, achipa <attila.cs...@gmail.com> wrote:
> OK, I just realized I can't attach files to postings. So, until I
> figure out what am I not seeing in the google groups interface, the

> patch can be downloaded fromhttp://www.atombiztos.hu/cron-2009-01-08.patch

[mdipierro]

The link does not work. Could you email it to me?

Massimo

On Jan 8, 10:49 am, achipa <attila.cs...@gmail.com> wrote:

> OK, I just realized I can't attach files to postings. So, until I
> figure out what am I not seeing in the google groups interface, the

> patch can be downloaded fromhttp://www.atombiztos.hu/cron-2009-01-08.patch

[achipa]

Hmmm, I sent you the patch directly last night (after your 5:51pm)
post, and the link works for me... Too much romulan ale ? I'll resend
anyway :)

bear@odin:/var/tmp$ wget http://www.atombiztos.hu/cron-2009-01-08.patch
--11:50:13-- http://www.atombiztos.hu/cron-2009-01-08.patch
=> `cron-2009-01-08.patch'
Resolving www.atombiztos.hu... 195.70.37.92
Connecting to www.atombiztos.hu|195.70.37.92|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 13,045 (13K) [text/x-diff]

100%
[=======================================================================================================================================================================>]
13,045 --.--K/s

11:50:13 (226.48 KB/s) - `cron-2009-01-08.patch' saved [13045/13045]