Starving queue with django-celery and rabbitmq

[Rogerio Carrasqueira]

Hello guys!

I think it’s kind of off-topic, because it’s basically I think more
from RabbitMQ than from Django itself, but I believe someone has
already may have gone through with a similar situation.

I have a Django application that runs with celery 3.1 and rabbitmq
3.8.2, It uses rabbitmq to distribute messages among a series of
workers who do several tasks.

It happens that at on a given moment when a quantity of tasks enters
very large in a given queue, it seems to me that that queue wants to
take all workers for itself. It is as if this task gained an absurd
priority in the system and the celery determined that this queue got
all the attention in the world and the workers worked just for it.

Faced with this scenario, absurd situations start: which I have a
worker who works in queue1, a worker b who works in queue 2, and I
have configured in the celery routes task x should be allocated in the
queue for worker b to execute. Entering an absurd amount of messages
in queue 1, worker a and b starts to working only at queue1, and the
queue2 is put aside. Only when giving an execution error of worker b,
the task that was in queue 1 is allocated in queue 2, occurring hell
in the system, leaving a mass about the organization of queues,
causing a series of bottlenecks in the system.

So, I ask friends for a light and put it down as I am configuring my
celery settings:

BROKER_TRANSPORT_OPTIONS = {}

CELERY_IMPORTS = ("core.app1.tasks", "core.app2.tasks")
CELERYD_TASK_TIME_LIMIT = 7200

CELERY_ACKS_LATE = True
CELERYD_PREFETCH_MULTIPLIER = 1
#CELERYD_MAX_TASKS_PER_CHILD = 1

CELERY_TIMEZONE = 'America/Sao_Paulo'
CELERY_ENABLE_UTC = True

CELERYBEAT_SCHEDULE = {
'task1': {
'task': 'tasks.task1',
'schedule': crontab(minute='*/30'),
},
'task2': {
'task': 'tasks.task2',
'schedule': crontab(minute='*/30'),
},
}

CELERY_RESULT_BACKEND =
"redis://server-1.klglqr.ng.0001.use2.cache.amazonaws.com:6379/1"
CELERYBEAT_SCHEDULER = 'redbeat.RedBeatScheduler'
CELERYBEAT_MAX_LOOP_INTERVAL = 5

CELERY_DEFAULT_QUEUE = 'production-celery'
CELERY_SEND_TASK_ERROR_EMAILS = False

CELERY_ROUTES = {

'tasks.task_1': {'queue': 'queue1'},
'tasks.task_2': {'queue': 'queue2},

}

Supervisor settings:

[program:app_core_production_celeryd_worker_a]

command=/usr/bin/python manage.py celery worker -n worker_a%%h -l INFO
-c 30 -Q fila1 -O fair --without-heartbeat --without-mingle
--without-gossip --autoreload --settings=core.settings.production
directory=/home/user/production/web_app/app
user=user
numprocs=1
stdout_logfile=/home/user/production/logs/celeryd_worker_a.log
stderr_logfile=/home/user/production/logs/celeryd_worker_a.log
autostart=true
autorestart=true
startsecs=10
stdout_logfile_maxbytes=5MB

[program:app_core_production_celeryd_worker_b]

command=/usr/bin/python manage.py celery worker -n worker_b%%h -l INFO
-c 30 -Q fila2 -O fair --without-heartbeat --without-mingle
--without-gossip --autoreload --settings=core.settings.production
directory=/home/user/production/web_app/app
user=user
numprocs=1
stdout_logfile=/home/user/production/logs/celeryd_worker_b.log
stderr_logfile=/home/user/production/logs/celeryd_worker_b.log
autostart=true
autorestart=true
startsecs=10
stdout_logfile_maxbytes=5MB

Thanks so much for your help!

Rogério Carrasqueira

[Rogerio Carrasqueira]

someone with a charitable soul could help?

Rogério Carrasqueira



Em qui., 25 de jun. de 2020 às 17:12, Rogerio Carrasqueira
<rogerio.ca...@gmail.com> escreveu: