Dump/restore (with regards to delayed messages)

[Kirill Gagarski]

Currently I'm doing major RabbitMQ upgrade (from quite old 3.x release) and my strategy is currently is to:

 - stop the producers

 - let the consumers consume the remaining messages

 - drop the old cluster

 - start from scratch

While it's fine in my case, it becomes tricky for delayed messages (both delayed retires and  delayed publish) , specifically "let the consumers consume the remaining messages" part.

It seems to be rather impossible to get the delayed messages earlier than their scheduled time (either to consume them earlier or to stash them somewhere else for the time of the upgrade). Shovel plugin does not seem to help much, it shovels using regular get operations, as I understand.

Of course, you may argue, "just update major versions one-by-one, you can even do it on a live cluster", but there comes this:
> Erlang 28 is only supported for brand new clusters

Eventually Erlang 27 will stop being supported and that will force the user to create a brand new cluster which will cause struggling with delayed messages.

I wanted to ask if there is a reliable update strategy that will preserve the messages. Like SQL databases provide you dump/restore machinery which mostly work across any versions (dump can even be a plain SQL script). Is there something like that for Rabbit? Or can dumping of delayed messages be done using reasonable effort (writing regular client code or using shovels). 

[Michal Kuratczyk]

Hi,

The note "Erlang 28 is only supported for brand new clusters" applies to specific patch releases, it's not a universal truth.

RabbitMQ 4.3.7 and 4.4.0 fully support Erlang 28, including a rolling upgrade from older RabbitMQ/Erlang versions.

Assuming you are talking about messages delayed using https://github.com/rabbitmq/rabbitmq-delayed-message-exchange,

that plugin has never been officially supported and is now discontinued altogether. Starting with RabbitMQ 4.4, quorum

queues can delay messages without any additional plugins or configuration:

https://www.rabbitmq.com/blog/2026/10/20/rabbitmq-4.4-release#delayed-messages

I'm not aware of a specific tooling to "prematurely consume" delayed messages from the old plugin, but in this day

and age, I'm fairly sure any LLM can give you a tool like that on the first try. The plugin uses DETS 

(https://www.erlang.org/doc/apps/stdlib/dets.html) for storing messages, which is effectively a simple database with an Erlang API.

Best,

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/rabbitmq-users/5836bb30-bfa6-42ac-bded-be7c9afd0198n%40googlegroups.com.

--

Michal

RabbitMQ Team

[Kirill Gagarski]

Hi Michal, thanks for your answers.

> The note "Erlang 28 is only supported for brand new clusters" applies to specific patch releases, it's not a universal truth.

RabbitMQ 4.3.7 and 4.4.0 fully support Erlang 28, including a rolling upgrade from older RabbitMQ/Erlang versions.

That is clear, thanks!

> Assuming you are talking about messages delayed using https://github.com/rabbitmq/rabbitmq-delayed-message-exchange

I am talking about modern delayed messages, delayed either by 4.4's quorum queues delay or 4.3's delayed retries. LLM gave me some Erlang script to execute using eval, but I assume it uses private internals, not guaranteed to work later and may be applicable only to one of these two cases.

[Michal Kuratczyk]

Hi,

You can defer a message with a token and then request messages with that token.

However, if you haven't set a token, there's no way to allow consumption of those messages.

We've added this as a feature request for the future.

Best,

To view this discussion visit https://groups.google.com/d/msgid/rabbitmq-users/f52ca7f7-2c8a-4da1-9d34-69142303be83n%40googlegroups.com.

--

Michal

RabbitMQ Team

[Kirill Gagarski]

Hello,

>You can defer a message with a token and then request messages with that token.

Thanks for clarifying. As I understand it, that helps when a consumer explicitly defers a message with a token, but it does not let us retrieve messages held by native delayed retry.

I wanted to ask a broader question: could RabbitMQ eventually offer a reasonably portable logical dump/restore mechanism for broker state and queued messages?

I understand that a cold filesystem backup preserves messages. From the upgradeability table, though, my impression is that restoring such a backup gives us a limited window—perhaps a few releases—rather than a path to an arbitrarily newer RabbitMQ version. That is an observation, not a claim of a fixed compatibility guarantee.

Some of us, ourselves included, occasionally leave an installation on an old release for too long. Upgrading it then means finding and running multiple intermediate RabbitMQ and Erlang versions. Containers can help package those versions, but adopting container tooling solely for an upgrade adds to the burden if you do not already use it. Containers also share a kernel, so sufficiently old software may still present compatibility problems.

When messages are short-lived, we have a workaround: stop producers, let consumers drain the queues, and start a fresh cluster. With TTL-based delay queues, we still have some room for manual intervention—we can fetch messages and republish them immediately or save them elsewhere.

Broker-managed delayed messages make that approach less general. Messages held for delayed retry or delayed publishing may not be available to ordinary consumers before their scheduled time, so we cannot necessarily drain or extract everything before replacing the cluster.

PostgreSQL provides a useful comparison. A logical dump often offers a practical route from an old installation to a fresh one without upgrading its physical storage through every intermediate version. Could RabbitMQ offer something comparable, or are there fundamental properties of messaging that make a portable export/import format impractical?

Best regards,
Kirill

[Michal Kuratczyk]

Technically it could, of course, and we hear this request from time to time. Historically

RabbitMQ stored almost exclusively transient messages. We had no stream support

and no delayed messages, so such a mechanism was not really needed.

Gradually, as we add features that store messages for longer, this indeed becomes more important.

Hopefully, not for upgrades - people really should upgrade their software from time to time. Those

who want to avoid it as much as possible can take advantage of the commercial extended support,

so they can upgrade less often. But there are still occasional migrations where this could be useful.

I won't make any promises about if/when we would add such a feature though. Delayed messages

in quorum queues are the first situation where messages are hard to consume. Everything else

can be shoveled to a new cluster or stored to a file or somewhere (there's plenty of tools

like Bento, where you can define a flow such as AMQP->file and file->AMQP, so you can extract

and load data).

How much are your messages delayed for? We assume the vast majority of users delay messages for

no more than a few days. In such case, just leaving the old cluster running until it's empty, should be

a decent workaround for now. It certainly should not prevent anyone from adopting a new version.

Best, 

To view this discussion visit https://groups.google.com/d/msgid/rabbitmq-users/cfea87a3-eeba-425b-aec1-1feced27fda8n%40googlegroups.com.

--

Michal

RabbitMQ Team

[Kirill Gagarski]

> Technically it could, of course, and we hear this request from time to time. Historically RabbitMQ stored almost exclusively transient messages. We had no stream support and no delayed messages, so such a mechanism was not really needed.

Yes, I can imagine I'm not the first to ask for this, and your point about transient messages is totally fair.

> people really should upgrade their software from time to time

No doubt about that, but not all of your users are perfect :) Postponing an upgrade tends to make the next one more painful, and the prospect of that pain makes it even easier to postpone it further. Though I must admit this is a common software maintenance problem, not something unique to RabbitMQ :D

> Those who want to avoid it as much as possible can take advantage of the commercial extended support, so they can upgrade less often. But there are still occasional migrations where this could be useful.

Commercial extended support lets you enjoy life a little longer before you eventually have to upgrade. But when that day comes, you still have to juggle multiple RabbitMQ versions and their runtime dependencies, potentially including different kernel versions in the worst caseю Admittedly, the kernel example is an extreme case, I promise to stop bringing it up now :). I'm not familiar with the details of the commercial support offering, but I doubt it magically allows you to skip intermediate upgrade steps. Please correct me if I'm wrong.

> How much are your messages delayed for? We assume the vast majority of users delay messages for no more than a few days. In such case, just leaving the old cluster running until it's empty, should be a decent workaround for now. It certainly should not prevent anyone from adopting a new version.

As I mentioned, I'm upgrading from an ancient version, so I'm not using native delayed messages or delayed retries yet. For now, we rely on the good old retry-queue pattern with message TTLs.

However, when it comes to progressively increasing retry delays (which native delayed retries provide), this pattern gets rather ugly. You end up needing multiple queues because of head-of-line blocking: with per-message TTLs, a message with a longer TTL can prevent a later message with a shorter TTL from being retried on time. So you eventually end up with something like a queue per TTL value.

The removal of mirroring support for classic queues makes things even more complicated. With classic queues, you could at least use `x-expires` to automatically clean up unused retry queues. This makes switching to native delayed retries very tempting, but the upgrade concern might become a blocker. With ordinary retry queues, you can at least shovel messages elsewhere, even if you have to do it manually.

As for leaving the old cluster running for a few days, I don't think it's quite that simple. To actually drain it, you need to stop publishing new messages to it. That means either stopping the relevant parts of your applications for several days or making them work with two clusters during the transition.

And making applications handle two clusters correctly might introduce even more complexity than a proper step-by-step upgrade.

Thanks for being understanding and for taking the time to explore possible workarounds!

[Michal Kuratczyk]

Hey,

We'll add something for the QQ deferred/delayed messages, at least a CLI command to make all those messages consumable,

so that draining queues is easy. As for the bigger "export/import" tool, I make no promises, but I have duly noted it.

> As for leaving the old cluster running for a few days, I don't think it's quite that simple. To actually drain it, you need to stop publishing new messages to it. That means either stopping the relevant parts of your applications for several days or making them work with two clusters during the transition.

You can switch the app over to the new cluster as if the old cluster was already drained. Leave the old one running, add some shovels to it,

so that once messages become consumable, they are shoveled to the new cluster and that's it.

> No doubt about that, but not all of your users are perfect :) Postponing an upgrade tends to make the next one more painful,

> and the prospect of that pain makes it even easier to postpone it further.

I know that's the case, but in this day and age, that's just professional malpractice. To ease this pain, starting with 4.0, we allow "jump upgrades".

You can go from 3.13 directly to 4.2 and you'll be able to upgrade 4.2 to 5.0, which is to be released next year. So in two steps, you can upgrade

from a release from early 2024 to a release from 2027.

Best,

To view this discussion visit https://groups.google.com/d/msgid/rabbitmq-users/84b6b942-e6c7-4105-ba36-2e558184bf1bn%40googlegroups.com.

--

Michal

RabbitMQ Team