Celery crashing with PRECONDITION_FAILED - delivery acknowledgement timed out (RabbitMQ 3.13.2)

[Vikram Bhimbar]

Hi,

We are using Celery with RabbitMQ as the broker. Our Celery workers consistently crash after ~30 minutes with the following error:

[CRITICAL/MainProcess] Unrecoverable error:
PreconditionFailed(406, 'PRECONDITION_FAILED - delivery acknowledgement on channel 1 timed out.
Timeout value used: 1800000 ms', ...)

RabbitMQ logs:

missed heartbeats from client, timeout: 120s

precondition_failed: delivery acknowledgement on channel 1 timed out.
Timeout value used: 1800000 ms

Setup & Configuration

• RabbitMQ version: 3.13.2
• HA setup using classic_queue_mirroring
• Celery settings (current):
  • task_soft_time_limit < 30 min
  • task_time_limit < 30 min
  • worker_prefetch_multiplier = 1
  • concurrency = 8

         Earlier we had:

• Prefetch multiplier = 4
• Autoscaling enabled
• Time limits ≥ 30 min
  
  Observed Behavior
• In the customer environment, some messages remain indefinitely in messages_unacknowledged.

• These messages do not appear in:

  celery -A <app> inspect reserved
• However, RabbitMQ still shows them as unacknowledged.
• After ~30 minutes, RabbitMQ triggers the ack timeout, leading to channel closure and Celery worker crash.

       In our local environment, the same configuration works fine and:

• inspect reserved correctly shows pending tasks
• No stuck unacknowledged messages

 Hypothesis

It seems that in the customer environment:

• Some tasks are fetched by Celery but never executed or acknowledged
• These “stuck” deliveries hit RabbitMQ’s consumer ack timeout (30 min)
• This results in PRECONDITION_FAILED and worker crash

Questions

1. What could cause messages to remain in messages_unacknowledged but not show up in celery inspect reserved?
2. Could missed heartbeats (seen in logs) be related to workers being blocked or stuck?
3. Are there known issues with classic_queue_mirroring and unacked messages in RabbitMQ 3.13.x?
4. Any recommended debugging steps to identify why Celery is not acknowledging these messages?

[Mirah Gary]

There are many known issues with classic queue mirroring, which is why they have been deprecated for years and were removed in RabbitMQ 4. 

RabbitMQ 3.13 has been out of community support since 2024. If you have a support contract, please open a ticket through the standard support channels. If you would like to open a support contract for extended support of RabbitMQ 3.13, please contact contact-tan...@broadcom.com. Otherwise, only RabbitMQ 4.2 and 4.3 are eligible for community support.

[Terry Rinck]

I would very carefully inspect your worker code and try to run with debug logging in the client's prod env. Without more context it's difficult to say what's happening here, but my instinct is that something has quietly crashed therefore sending neither an Ack or NAck back to the broker but also not raising a channel or connection error, thus seeing the ' messages_unacknowledged' count remain and only getting the PRE failed on consumer timeout.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/rabbitmq-users/a6f4193b-a3ea-4ddc-8498-1019a0a3c9a0n%40googlegroups.com.

[Vikram Bhimbar]

Hi Terry,

Can you please let me know what exactly in the code you are expecting from me when you say "I would very carefully inspect your worker code and try to run with debug logging in the client's prod env."?
Or you are suggesting we should enable debug on production environment?

[Terry Rinck]

Right, I'm suggesting the debug logging could help point to bits of the client code that seem to be dropping the messages. Or at the very least help rule out certain other issues. In my experience messages stuck 'unacknowledged' suggests miss handling somewhere along the consumers message processor flow.

To view this discussion visit https://groups.google.com/d/msgid/rabbitmq-users/ab1b7537-0f24-4fa4-ab3d-7cbcb1639487n%40googlegroups.com.

[Vikram Bhimbar]

Hi Terry ,

Thanks, we will do that and update on the forum.

[Luke Bakken]

Do yourself, your coworkers, and your employer a favor and stop using unsupported, out-of-date software.

Every minute you spend investigating this issue rather than upgrading RabbitMQ and Celery, and using quorum queues, is wasted time.

And yes, Terry is correct - the consumer timeout message means you have one or more applications / celery workers that crash while they have a message delivered to them, but don't close the channel/connection.