Celery and gevent limits?

52 views
Skip to first unread message

Mitch Raful

unread,
Jul 31, 2020, 3:39:36 PM7/31/20
to Django users
I have a django app that uses celery to do lots of polling and celery beat to schedule those.  I use gevent instead of worker processes.  It polls network devices using various things such as a vendor api, snmp, and, if needed, ssh to the command line.  

What I think I am seeing is as follows.  It looks like a long running task kicks off, apparently running quickly.  However, when another tasks kicks off the original is interrupted.  Once the interrupting task has finished or momentarily stops, the original task picks back up.  Tasks that I have timed to take thirty to forty minutes stand alone can take up to two hours under celery with gevent.

This leads me to wonder if I have reached a limit with using Celery/Gevent or am I doing something wrong with the way I have set up the tasks with app.task?

Thanks,

Mitch
Reply all
Reply to author
Forward
0 new messages