dynamic task scheduling + django

[oz katz]

Hello Django gurus,

I'm working on a django application, in which for every user, a RESTful web-service request needs to be called in the background (to another server in the organization)  - 

every X minutes (configurable by the user, defaults to 5 minutes). the response from the web-service call might trigger a notification -

an e-mail to the user (depending on the result), as well as be registered in the DB using django's model API.

What solutions are best for scheduling this type of events?

I've been looking at celery but I can't figure out how to dynamically add or change scheduled tasks.

how can I create a new task when a user is registered? how do I change the timing when a user changes his settings?

Your advise will be highly appriciated! :)

Thanks,

Oz

[uriel katz]

what about this simple solution:
write a deamon that has a dict with a mapping between interval (5,10...etc minutes) to a list of userid(or whatever user info you want to store) then create a a thread that sleep for a minute then increment a counter then you go over the dict keys and do:
for interval,user_ids in intervals.items():
  if counter % interval == 0:
     process(user_ids)

also start a http server on the deamon then you can get a RESTful API on the deamon to add/change schedules :)

also don`t worry for counter overflow,a integer will give you 2^31 minutes :)

--
You received this message because you are subscribed to the Google Groups "PyWeb-IL" group.
To post to this group, send email to pywe...@googlegroups.com.
To unsubscribe from this group, send email to pyweb-il+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/pyweb-il?hl=en.

--
-Uriel Katz

[oz katz]

Hi,

thanks for the idea. but i'm not sure I want to start re-inventing the wheel here.

I see 2 problems with this solution:

1. thread management: every 5 minutes (the default) I will have X threads started in the process (where X = number of users whose interval is 5 minutes). all of which are trying to access the same web service. afterwhich, the system will be mostly idle for the rest of the time (until the next 5 minutes interval).

2. how would you scale such a solution? Maybe horizontal sharding of the users's table?

that's something you will have to maintain, as opposed to celery that allows adding more

worker servers in a pretty transparent way (from what I understood from the docs, anway - since I have no real experience with it).

Thanks again :)

Oz

[uriel katz]

in the most simple way,use a queue to put the user id and other threads are the workers this way you limit the number of concurrent access (by limiting the number of working threads) and also you a queue server(plenty of those available) and start workers on another machine this way you achive horizontal scalability.

or you can use gevent,put a gevent.Pool() (for concurrency management) and it will scale to more than you will probably need in one machine in one thread :)

[Natan Brosztein]

Hi there,

I am not a Django guru but a very simple solution is, if you are working in Linux, schedule a CRON job that triggers every minute. This job uses a very simple python script that checks a common data store and sends the notifications as needed.

Hope this helps

Natan

On Sun, Aug 22, 2010 at 6:18 PM, oz katz <ozka...@gmail.com> wrote:

--

[Zohar Arad]

Oz,
It sounds like a classic case of Message Queues - Try looking into
ActiveMQ or RabbitMQ (and Gearman to a lesser extent)

> > pyweb-il+u...@googlegroups.com<pyweb-il%2Bunsubscribe@googlegroups.c om>
> > .

[oz katz]

Hi Guys,

thanks for your input! i decided to go with a combination of an MQ broker (rabbitMQ using amqplib)
and the multiprocessing library. since i'm only farmiliar with IBM's MQ solution, there is a bit of a learning curve
for AMQP (exchanges, routes, binding, etc). but it's quite interesting actually :)

anyway - thank you guys for the help!
Oz

To unsubscribe from this group, send email to pyweb-il+u...@googlegroups.com.