Are lazy queues still bound by memory limit?

[Steve Suehs]

I've been experimenting with Clojure and RabbitMQ using the langohr library.  Thank you for building and sharing it.

I've been learning more about the care and feeding of a rabbitmq server.  I crashed mine installed on my dev laptop several times by filling a queue with messages.  If queues fill up and memory gets tight, a server can be overwhelmed and fall over.  If I set or arrange for a reasonable memory limit, things go ok. I am now running RabbitMQ in a Docker container with a memory limit.

My latest experiments were with lazy queues, which are supposed to write events to disk and not exhaust memory.  I still seem to hit..well, not hit, rather, an asymptotic limit at around 261 million events in the queue.  Is this considered outrageously and unreasonably large?  Publication rate slowed to 700/s.

I also notice that if the consumers are caught up with the producer everything seems to scream at twice the rate than when storage is used. If storage is used I'm seeing about 6000r/s.  That's cool...just not exactly what I expected from the description of a lazy queue.

[Michael Klishin]

This is a question for rabbitmq-users but I'm happy to answer it here.

Lazy queues per se don't have a limit but RabbitMQ message store still does: when messages

are sent to the message store [as opposed to being embedded into queue index], message store index(es)

come into play and by default the index is in-memory.

There is a plugin [1] that lets you use LevelDB for the index. Even though LevelDB is an efficient

data store, expect a write throughput hit compared to the default implementation.

1. https://github.com/rabbitmq/rabbitmq-msg-store-index-eleveldb

--
You received this message because you are subscribed to the Google Groups "clojure-rabbitmq" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure-rabbitmq+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Michael Klishin]

The point of lazy queues is not that nothing ever is stored in memory. It's that queues try to move messages

to disk very aggressively (default mode keeps a portion in RAM using metrics such as ingress/egress rates and various

configurable values).

[Steve Suehs]

Thank you!

To unsubscribe from this group and stop receiving emails from it, send an email to clojure-rabbit...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ