RabbitMQ Persistence

[Ashutosh R]

Hi,

I am using RabbitMQ for my application. I had a few questions regarding RabbitMQ persistence.

1. As per what I know, RabbitMQ uses Erlang Mnesia for persistence. How can I view the persisted messages?
2. I suppose Mnesia can hold data upto 4GB. I am not sure on this number but how does RabbitMQ manage archival of persisted messages. Is there a configuration to be added/modified to manage this in RabbitMQ?
3. What is the impact on performance of RabbitMQ when persisted message reaches threshold?
4. Does RabbitMQ persists all the messages in a separate file?

[Michael Klishin]

On 6 February 2016 at 01:25:49, Ashutosh R (rava...@gmail.com) wrote:
> As per what I know, RabbitMQ uses Erlang Mnesia for persistence.
> How can I view the persisted messages?

Messages and message indices  are not stored in Mnesia. Metadata (vhosts, users, permissions, queues,
exchanges, bindings, runtime parameters/policies) is.

> I suppose Mnesia can hold data upto 4GB. I am not sure on this number
> but how does RabbitMQ manage archival of persisted messages. 

Mnesia doesn't have this limit AFAIK, a single DETS file does but Mnesia doesn't
have to be limited to a single file ;)

See https://github.com/rabbitmq/internals. It has some relevant information.
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Ashutosh R]

Thanks for the quick reply.

So if messages are not stored in Mnesia, where are the messages persisted by RabbitMQ?

How can I view all the messages sent to the queue?

Does RabbitMQ persists all the messages in a separate file?

[Michael Klishin]

On 6 February 2016 at 01:49:20, Ashutosh R (rava...@gmail.com) wrote:
> So if messages are not stored in Mnesia, where are the messages
> persisted by RabbitMQ?

To a node-local message store.

> How can I view all the messages sent to the queue?

There is no tool that lists messages, although there is an experimental CLI tool
that can load messages from disk and publish them (à la Shovel).

> Does RabbitMQ persists all the messages in a separate file?

In many files. See the internals repo and

https://github.com/rabbitmq/rabbitmq-server/blob/master/src/rabbit_msg_store.erl
https://github.com/rabbitmq/rabbitmq-server/blob/master/src/rabbit_msg_file.erl
https://github.com/rabbitmq/rabbitmq-server/blob/master/src/rabbit_msg_store_ets_index.erl
https://github.com/rabbitmq/rabbitmq-server/blob/master/src/rabbit_msg_store_gc.erl
https://github.com/rabbitmq/rabbitmq-server/blob/master/src/rabbit_queue_index.erl

[Ashutosh R]

Let me put it this way. I have RabbitMQ server running on Linux. What location I can check the persisted messages?

Are the messages persisted even after they are picked up from queue?

How long the messages are persisted on the node-local message store?

[Michael Klishin]

On 6 February 2016 at 02:56:51, Ashutosh R (rava...@gmail.com) wrote:
> What location I can check the persisted messages?

There are two message stores (for persistent and transient messages)
under RABBITMQ_MNESIA_BASE: http://www.rabbitmq.com/relocate.html

which on Linux is typically under /var/lib/rabbitmq by default.

> Are the messages persisted even after they are picked up from
> queue?

Messages that are consumed and acknowledged are deleted.

> How long the messages are persisted on the node-local message
> store?

See above. If you want a stream that's stored on disk and can be consumed
repeatedly ("replayed"), see Kafka.

[Ashutosh R]

Thanks Michael!

[Ashutosh R]

One more thing. I am running into the following problem.

Message is being sent to the queue but not yet consumed. For some reason the broker (RabbitMQ server) goes down. After  the server is back up and running, I do not see the messages that were queued.

I thought RabbitMQ persists messages and the messages are queued once the server is up.

[Ashutosh R]

Queue is durable and message is sent with delivery mode PERSISTENT.

[Michael Klishin]

On 6 February 2016 at 04:09:45, Ashutosh R (rava...@gmail.com) wrote:
> Queue is durable and message is sent with delivery mode PERSISTENT.

We cannot suggest much without seeing your code and log files. We see auto-delete
and exclusive queues trip up beginners all the time (to the point we will
be logging all related events in 3.7.0: https://github.com/rabbitmq/rabbitmq-server/issues/590).

I'm not sure if "for whatever reason" is an example or an observation but for cases with
messages coming in all the time without consumers, you probably want to use lazy queues:
http://rabbitmq.com/lazy-queues.html (which is an implementation detail and orthogonal to durability
but still). 

[Ashutosh R]

As far as I know, auto-delete is turned off. Using lazy queues might feel up the disk soon. What is your take on that?

I have a dumb query but want to be clear on what king of queue and exchange works for me. I have a queue which is bind to exchange with some binding key. The queue is connected to 3 consumers. I want all the consumers to listen to all the messages coming to the queue. I read somewhere that messages are delivered to consumers in round robin fashion. But I want all messages be delivered to all consumers. Please suggest.

[Michael Klishin]

 On 9 February 2016 at 22:51:15, Ashutosh R (rava...@gmail.com) wrote:
> The queue is connected to 3 consumers. I want all the consumers
> to listen to all the messages coming to the queue. I read somewhere
> that messages are delivered to consumers in round robin fashion.
> But I want all messages be delivered to all consumers. Please
> suggest.

A message can only be delivered to one consumer at a time. You want N queues
with 1 consumer each and an exchange that can route to more than one queue
(typically fanout or topic).

[Ashutosh R]

Thanks Michael!

Coming back to persistence, the queue I have setup is durable and auto delete is set to No. Also the messsage sent was PERSISTENT. I created the queue using Management Interface. But when the queue was shutdown by me and started again the message which was undelivered did not requeue. Any other pointers? 

[Michael Klishin]

On 9 February 2016 at 23:19:27, Ashutosh R (rava...@gmail.com) wrote:
> Coming back to persistence, the queue I have setup is durable
> and auto delete is set to No. Also the messsage sent was PERSISTENT.
> I created the queue using Management Interface. But when the
> queue was shutdown by me and started again the message which was
> undelivered did not requeue. Any other pointers?

I find it very difficult to believe that this is some kind of bug. We would be getting
reports for something like this multiple times a day if it was.

I cannot suggest much without seeing your code and a step-by-step transcript
that can be used to reproduce. 

[Ashutosh R]

Worked fine now. I do not know why it failed before. Thanks!

How can I configure the threshold of unacknowledged messages a queue can hold? Is there a default threshold value?

[Michael Klishin]

On 10 February 2016 at 02:20:06, Ashutosh R (rava...@gmail.com) wrote:
> How can I configure the threshold of unacknowledged messages
> a queue can hold? Is there a default threshold value?

There is no limit on unacknowledged deliveries.
 You only can set total queue limit and message TTL:

https://www.rabbitmq.com/maxlength.html
http://www.rabbitmq.com/ttl.html

[Ashutosh R]

I was going through the RabbitMQ Access Control.

https://www.rabbitmq.com/man/rabbitmqctl.1.man.html#Access%20control

Few things need help with

1. I want to create a user which can only be used by the application acting as Producer and Consumer. What is the permissions string I set when using 

set_permissions [-p vhostpath] {user} {conf} {write} {read}

2. I use Spring RabbitMQ for our application. In Spring application context, I create RabbitMQ connection factory in the following manner. I wanted to check if there is a way to avoid adding password in the context file OR some way to pass it in encrypted way so that anyone accessing context file does not get real password.

<rabbit:connection-factory id="connectionFactory" host="localhost" port="5672" virtual-host="/" username="[username]" password="[password]" />

Thanks!

[Michael Klishin]

On 11 February 2016 at 01:07:45, Ashutosh R (rava...@gmail.com) wrote:
> Few things need help with

Ashutosh,

Would you mind starting new threads for new questions? It's not very cool to stuff
existing ones with more and more questions. Thanks. 

[Ashutosh R]

Sure