A question about scale of periodic tasks

[Harel Malka]

Hi
Looking at the latest development of periodic tasks, I can see that I
can now set celerybeat to use a DatabaseScheduler, which will use a
bunch of database tables to call up tasks, as opposed to the way i
used to create periodic tasks so far, by using decorated tasks etc.
I have a scenario where I have a periodic task which runs every few
minutes, queries the database for various actions each with a
scheduled time (cron like) and executes those tasks based on their
schedule, last execution time etc. In a sense I'm manually doing what
appears to be the same functionality celerybeat is now doing with
DatabaseScheduler.

The only thing is that I have tens of thousands such schedules set to
run at various intervals and more being set at the will of users.

My question is can I create database entries using the
DatabaseScheduler models, modify or remove them at will and have
celery beat execute the tasks without me having to constantly manage
that?
Can celerybeat scale up to a few tens of thousands such periodical
tasks? Or am I better off staying where I am, running one task every
few minutes collecting my own data on the fly from the database to
determine which tasks to run and with what parameters.

Thanks,
Harel

[Philip Gatt]

Harel,

I'm not a celery developer and I've only been using it for 6 months. I
LOVE it, but I am getting the feeling not many people have used it in
high traffic situations. I think the main idea of celery running tasks
in the background instantly (like calling .delay() or .apply_async
with a short countdown) works fine, but I hit a problem scheduling too
many future tasks. You can read about it here:
http://groups.google.com/group/celery-users/browse_thread/thread/f10fa4c0858289f6

Celerybeat looked really cool, but I think celerybeat was still beta
when I was developing my project so I decided to stay clear of it. I
wanted would to get rid of my 30 line long crontab though, so I asked
this question on SO: http://stackoverflow.com/questions/2367824/storing-crontab-file-inside-my-project

Best Regards,
Philip

Note to celery community: I apologize in advance if my assumptions
about celery are wrong. Please show me if I'm wrong about something.

[Harel Malka]

Thanks Philip. So I see you hit a limit of 65K periodical task.
Although I have something in the region of 30,000 periodical events of
different schedules and cron settings I will hit 65K soon enough. So
if there is a limit - its not it for me. Is this also happening using
the DatabaseScheduler?
Harel

> --
> You received this message because you are subscribed to the Google Groups "celery-users" group.
> To post to this group, send email to celery...@googlegroups.com.
> To unsubscribe from this group, send email to celery-users...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/celery-users?hl=en.
>
>

[Philip Gatt]

I haven't tried the DatabaseScheduler but that sounds like a good
workaround for the problem. I bet it will work because the error came
from the ampq library.

[Harel Malka]

I am considering switching to it and I will post feedback, though I
need some input from Ask (Ask are you here?) regarding the scalability
of this solution in my scenario. I can' t afford failures as my
current solution works.
Harel

[Ask Solem]

On Tuesday, March 22, 2011 at 11:11 PM, Harel Malka wrote:

I am considering switching to it and I will post feedback, though I
need some input from Ask (Ask are you here?) regarding the scalability
of this solution in my scenario. I can' t afford failures as my
current solution works.
Harel

The limit applies to eta/countdown tasks, not periodic tasks.

There is no known limit to the number of periodic tasks, and the celerybeat

scheduler should perform well even with a large number of schedule entries.

[Harel Malka]

OK, that's good news. I'll kick off an experiment sometime this week
where periodic tasks can be scheduled by random users, and there would
be plenty of them.
I'll report back :o)
Thanks!
Harel

[Philip Gatt]

Great news about celerybeat scaling without problems. Will the workaround we mentioned (using a db scheduler) for scheduled tasks work to get us past the 65k limit?