Celery makes RabbitMQ use so much virtual memory that it crashes.

[Ludwig Schmidt-Hackenberg]

Hi, 

till now I was very pleased with celery. I use Celery to distribute task on my compute server. I started with ~10k tasks which worked fine. Now I moved to ~100k tasks and after some time RabbitMQ crashes with following error message:

eheap_alloc: Cannot allocate 1098556536 bytes of memory (of type "heap").

Aborted (core dumped)

I guess celery is creating too many queues, but I don't know which setting I sent wrongly.

This is my celeryconfig.py

from utilities import get_number_cpu_cores

# List of modules to import when celery starts.

CELERY_IMPORTS = (...)

BROKER_URL = 'amqp://guest:guest@localhost:5672//'

CELERY_RESULT_BACKEND = "amqp"

CELERY_TASK_SERIALIZER = 'json'

CELERY_ACCEPT_CONTENT = ['json']  # Ignore other content

CELERY_RESULT_SERIALIZER = 'json'

CELERY_TIMEZONE = 'Europe/Berlin'

CELERY_ENABLE_UTC = True

CELERYD_CONCURRENCY = get_number_cpu_cores() * 2

I tried to log what RabbitMQ is doing with 

 curl -i -u guest:guest http://localhost:15672/api/vhosts >> vhosts.log

 curl -i -u guest:guest http://localhost:15672/api/queues >> queues.log

This is the last entry of vhosts.log: 

Server: MochiWeb/1.1 WebMachine/1.10.0 (never breaks eye contact)

Date: Wed, 11 Jun 2014 19:03:34 GMT

Content-Type: application/json

Content-Length: 627

Cache-Control: no-cache

[{"message_stats":{"ack":59994,"ack_details":{"rate":0.0},"deliver":59994,"deliver_details":{"rate":0.0},"deliver_get":121603,"deliver_get_details":{"rate":0.0},"deliver_no_ack":1630,"deliver_no_ack_details":{"rate":0.0},"get":59979,"get_details":{"rate":0.0},"publish":118167,"publish_details":{"rate":0.0}},"messages":69099,"messages_details":{"rate":0.0},"messages_ready":69099,"messages_ready_details":{"rate":0.0},"messages_unacknowledged":0,"messages_unacknowledged_details":{"rate":0.0},"recv_oct":118595149,"recv_oct_details":{"rate":0.0},"send_oct":91945666,"send_oct_details":{"rate":0.0},"name":"/","tracing":false}]

The last entry of queues.log is cut off and the one before is 64MB. 

Tell me if you need more information!

Thanks!

[Ludwig Schmidt-Hackenberg]

Two things I forgot:

I use another rabbitmq instance to ensure that there are no more than number_of_cores*2 (one for each worker as defined in celeryconfig.py) tasks in the celery queue at any given moment. 

Also I collect the result of each task with get() before launching a new task.

Cheers

Ludwig

[Ask Solem]

Why are you using the amqp result backend?  If you want to do RPC then use the rpc result backend,

if you want persistent results use a database based result backend.

-- 
Ask Solem
twitter.com/asksol | +44 07454281208

[Ludwig Schmidt-Hackenberg]

Why are you using the amqp result backend?  If you want to do RPC then use the rpc result backend,

if you want persistent results use a database based result backend.

There is no particular reason. I just followed the tutorial: http://celery.readthedocs.org/en/latest/getting-started/first-steps-with-celery.html#keeping-results 

[Ludwig Schmidt-Hackenberg]

Hi Ask, 

I thought again about your comment using a database.

For me it seems using a database seems like a huge over head for my use case. I have a scheduler class which feeds Celery image processing tasks. These Celery tasks write their results to file and return the result file name. The scheduler collects the results as soon as they are finished and starts new tasks. So in this sense, I need persistent results, because I want to use them later. On the other hand the results only need to be stored till they are collected by the scheduler.

Thanks,

Ludwig

[Ask Solem]

What doi you mean by ‘use them later’?  What you are describing sounds like the classic RPC scenario.

RabbitMQ is not a key value store, using one queue per task is very expensive, the rpc backend is more suitable

using one queue per client.

[Ludwig Schmidt-Hackenberg]

> What doi you mean by ‘use them later’?  

With later I mean, that I want to collect the results of the celery worker as soon as the worker has finished. After I have collected the result (I use get()) I don't need Celery to keep the results anymore. 

> What you are describing sounds like the classic RPC scenario.

But isn't RPC meant for synchronous tasks? Sending one task and then waiting for the task to finish before continuing? I launch x tasks and then wait for one to be finished, collect the result, launch a new one and start waiting again for one of the x to finish.

> RabbitMQ is not a key value store, using one queue per task is very expensive, the rpc backend is more suitable using one queue per client.

I saw that Celery is creating a lot of queues, but thought, this is just the way it is. Can you make Celery only create one queue in RabbitMQ for all tasks?

Thanks