Messages Stuck in Unacked

[Andrew Kowalik]

Stack:

Heroku

Celery

Rabbitmq

On some of our queues we are using acks_late=True in Celery. When True the worker will not acknowledge until the job is completed. Since these workers are run on Heroku, its possible for the process to be killed while a job is running. When this happens, the message remains in unacked. I can even turn off all workers and the message is still in unacked status. Rabbitmq will report 0 consumers in this case. Trying to get these messages to be reset to allow for reprocessing.

Any thoughts? Thanks!

[Michael Klishin]

We don't cover Celery or Heroku add-on on this list.

As tutorial 2 [1]
should mention, dropped connections (technically: closed channels) will result in unacknowledged messages
being re-queued. Perhaps it takes a while to detect peer loss (shouldn't be if the process
is killed, but who knows), so take a look at using a lower (6-10 seconds) heartbeat timeout
value [2]. Otherwise I'd expect that Celery is trying to be too smart.

 Note that the number of consumers
is irrelevant for deciding whether to re-queue unacknowledged deliveries, all that matters is whether the channel (or its entire connection) is still open.

1. http://www.rabbitmq.com/getstarted.html
2. http://rabbitmq.com/heartbeats.html
--
MK

Staff Software Engineer, Pivotal/RabbitMQ