Running Scheduled functions in Cherrypy

[Rob Cowie]

Hi all,

Is there some way of extending cherrypy to include that ability to
schedule the execution of a given function say, every week?

I have a cherrypy app that provides access to a database with all the
usual CRUD and search functionality one would expect. The next thing I
would like to add is the ability to send weekly emails to a
predetermined address.

If anyone has any bright ideas about how best to acheive this I'd love
to hear them.

Cheers,

Rob C

[Christian Wyglendowski]

Hi Rob,

CherryPy doesn't provide anything for that directly. I think the best
way of doing this is with the recur.py module from the Dejavu project.

http://projects.amor.org/dejavu/browser/trunk/recur.py

It completely rocks for scheduled and recurring tasks. And it is
written by Robert Brewer, who is one of the main developers for
CherryPy. So there is your CP connection ;-)

Christian
http://www.dowski.com

[thakadu]

I have done this with a "daemon" thread in the following way:
import cherrypy
import thread
import time

def daemon():
while 1:
do_periodic_stuff()
time.sleep(seconds)

cherrypy.root = my_cherrypy_root()
thread.start_new_thread(daemon)
cherrypy.server.start()

In my "do_periodic_stuff" I am reading some updated information and
then inserting it into a database and updating a global data structure.
The cherrypy exposed methods are displaying information from the global
data structure. Depending on what you want to do you may need to use
some locks as well. I got away with not using any locks by taking
advantage of the GIL.

[Robert Brewer]

*blush* but yes, that is what I would have suggested, too. I use it in
production. Have a look at the Worker class in that module.

Dejavu includes that module, but it's designed and distributed as a
standalone. See http://projects.amor.org/misc/wiki/Recur

Robert Brewer
System Architect
Amor Ministries
fuma...@amor.org

[Rob Cowie]

Thanks, I'll take a look at it.

I was beginning to go mad trying to get the sched module to do what I
want. I've never used it before but it certainly seems to be one of the
weakest members of the python standard library.

I'll let you know how I get on with recur. One thing that I might need
some help with is where to incorporate it in my server. Should it be in
its own thread? etc.

Cheers,

Rob C

[Robert Brewer]

Each Worker object will start those separate threads for you when you
call its "motivate" method. :)

[Rob Cowie]

Well recur works a treat!

However.... I have one remaining problem.

On cherrypy server startup, I call a function that creates a custom
recur.Worker object and calls its .motivate() method. A new thread is
spawned and the worker does its thing when required.

I had assumed that when I kill the cherrypy server, all threads spawned
by it would be killed as well. This is not the case. So, I added a
function that should call the .stop() function of my worker object.

Is this enough to kill the thread? Currently, when I kill cherrypy at
the command line with cmd-C, I am not returned to a shell prompt as a
thread is still running.

I'd appreciate any help!

Rob C

[Robert Brewer]

It should be enough. Here's what I do (with a "workers" dict) that works
fine:

def unload_workers():
while workers:
name, worker = workers.popitem()
worker.stop()

Tracking down which thread is hanging can be an involved process. Have
you verified that your stop() call is actually being executed? (through
liberal use of log() statements or your favorite debugging tool ;)

[Kevin Dangoor]

On 3/29/06, Rob Cowie <cowi...@gmail.com> wrote:
> Is this enough to kill the thread? Currently, when I kill cherrypy at
> the command line with cmd-C, I am not returned to a shell prompt as a
> thread is still running.

I believe that thread will die when the timer fires.

Kevin

[Robert Brewer]

It shouldn't wait for that. The Worker's curthread attribute is actually a threading.Timer, which uses a threading.Event with a threading.Condition object (whew!). Anyway, Condition.wait sleeps for small periods of time, and can be interrupted. From the comments in threading.py:

# Balancing act: We can't afford a pure busy loop, so we
# have to sleep; but if we sleep the whole timeout time,
# we'll be unresponsive. The scheme here sleeps very
# little at first, longer as time goes on, but never longer
# than 20 times per second (or the timeout time remaining).

So the Worker's thread should die within 1/20 of a second at most.

[ingo...@gmail.com]

Robert Brewer wrote:

> Dejavu includes that module, but it's designed and distributed as a
> standalone. See http://projects.amor.org/misc/wiki/Recur

Off topic for Cherrypy, but related to the subject,

Is there a simple sample somewhere that shows a not so confident Python
user, like me, how to use the recur module?

I'd like to use it to make time lapse movies. For example, take an
image every 30 minutes between 7.00 and 17.00 on each work-day and none
in the weekend.

Ingo

[Rob Cowie]

I too stored references to my worker objects and iterated through them
calling .stop(). It didn't work.

I did verify that the stop() method was being called.

Since then - I have changed the way I instantiate the worker objects. I
had used:
'cherrypy.server.start_with_callback(scheduler.create_weekly_email)'.

Now I just add 'scheduler.create_weekly_email' to the
'on_server_start_list' and all is well. I'm not sure why that is the
case - possibly something to do with the callback being executed in a
thread separate from the server.

Ingo: you are welcome to take a look at the code I use. It is
relatively simple so might serve as a good example. If you want me to
email it - let me know.

Rob C

[IngoognI]

Rob Cowie wrote:

> Ingo: you are welcome to take a look at the code I use. It is
> relatively simple so might serve as a good example. If you want me to
> email it - let me know.
>

Thanks Rob, that would be great,

Ingo

[Kevin Dangoor]

Interesting to know that it polls 20 times a second *at a minimum*. I
never would've guess that it's that responsive to cancellations. One
probably wouldn't want too many of those threads going at once! (Which
is where a sched-based scheduler comes in handier.)

Kevin

[Robert Brewer]

Wow, that's complicated! ;) The recur module doesn't do compound
schedules; you'd have to combine those three requirements yourself,
either with nested workers, or with a single worker that just ignores
invalid datetimes. For example, I'd probably write:

class MovieWorker(recur.Worker):
def work(self):
if (self.nextrun.day not in (5, 6)
and (7 < self.nextrun.hour < 17)):
take_frame()

mw = MovieWorker("30 minutes")

You could also pass the (5, 6) and (7, 17) bounds as arguments to the
worker, if you needed that flexibility. Normal Python rules still apply.
;)

[IngoognI]

Robert Brewer wrote:
> ingo...@gmail.com wrote:

> > I'd like to use it to make time lapse movies.
>

> Wow, that's complicated! ;) The recur module doesn't do compound
> schedules;

Rob C.'s code gave me a head start, thanks for that Rob, and the
"first" thing I did was to crash into that wall ....

> you'd have to combine those three requirements yourself,
> either with nested workers, or with a single worker that just ignores
> invalid datetimes. For example, I'd probably write:

> [...]

.... and then you appear to push me in the right direction. I'll have
to 'schedule' a bit more Python-time this weekend.

Thanks,

Ingo

[Steve Zatz]

http://projects.amor.org/misc/wiki/Recur

Haven't been able to reach that URL. ? if project exists at another location.

[Robert Brewer]

Sorry about that. A router choked. Back now.