Node start crash loop

[aviv salem]

Hey... We're using rabbitmq 3.7.11, erlang 21.2.3, centos 7

3 node cluster, with ha-mode exactly 2.

We encountered the following situation on several occasions, on different rabbit deployments.

1. The cluster gets overloaded for some reason (our fault)

2. One of the nodes crashes due to out of memory (probably our fault) 

3. When this node tries to re-start, upon startup, it fills the memory until it crashes due to out of memory, and fails to start. (we can't do nothing about it)

4. This node cannot be raised again, unless we increase the memory of the machine, or erase the message store (and lose messages) .

My question is:

1. Am I doing something wrong?

2. What can we do differently to raise the node so it won't be stuck in a crash loop. 

3. Is there a way to not load ALL the message store to memory on startup? 

[Luke Bakken]

Hi Aviv,

Are you using lazy queues? I'm assuming that this scenario is caused by many messages remaining in a regular queue when the node starts.

Thanks -

Luke

[aviv salem]

Hey luke... 

All our queues are lazy... 

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Luke Bakken]

Hi Aviv,

We have some documentation with regard to lazy queues and your scenario: https://www.rabbitmq.com/lazy-queues.html#caveats-limitations

You should set queue_index_embed_msgs_below to 0 which will prevent this issue from happening, but it also won't help with your current situation, since the message bodies have already been added to the index file.

Thanks,

Luke

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

[aviv salem]

Thanks for the reply!

Is it just adding the line 

queue_index_embed_msgs_below=0

To the rabbitmq.conf file? 

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

[Luke Bakken]

That is correct. A restart is required after.

To the rabbitmq.conf file? 

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

[aviv salem]

Wait... After reading the documentation you sent, I'm a bit confused...

Only Messages below 4k should be embedded into the index... But our scenario had only messages which are larger then 4k... Is it possible, that during node startup, even messages which are larger the 4k gets loaded into the memory? 

To the rabbitmq.conf file? 

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

[Luke Bakken]

Hi Aviv,

I remember all of the testing we did to produce that doc. Yes, if all of your messages are larger than 4KiB and your queue was lazy when they were published, they should not be loaded when your node starts. I can re-test this at some point to confirm.

Thanks -
Luke

[Luke Bakken]

Hello again,

Please see this comment: https://github.com/rabbitmq/rabbitmq-server/issues/1284#issuecomment-355513904

Even with all messages exceeting 4KiB in size, some memory is required (up to 696 bytes per message) when lazy queues are loaded as Diana's comment mentions.

That GitHub issue is part of what informed the lazy queues documentation.

Thanks -

Luke

[aviv salem]

I read back a little on the comments to the issue you sent... 

It seems that my suspicion was correct And on startup, the node loads messages to memory, even if they're bigger then 4k (the 600 bytes is only if you set the embed_below value to 0)

Is this a bug? Or am I missing something?

Anyway, it seems that setting the embed value to 0 Will work for us. 

I'll give it a try and let you know how it goes... 

Thank you for your time! 

--

[Luke Bakken]

Hi Aviv,

I can't reproduce what you report. I created a queue named "test-lazy" (durable, lazy) then I published 65536 messages of 65536 bytes each via PerfTest using these arguments:

--predeclared --queue test-lazy --flag persistent --size 65536 --pmessages 65536 --producers 1 --consumers 0

The "test-lazy-65K-msgs.png" file shows the state of that queue after the above command completes.

I then restarted RabbitMQ. The other two files show the state of the queue and overview after the restart. You can see that no messages have been loaded into memory.

In your case, either the queue was not declared lazy prior to publishing those messages, or perhaps there is at least one small message in the data store that triggers loading into memory.

Thanks,

LUke

[aviv salem]

one small message is enough to load EVERYTHING into the memory?

--

[Luke Bakken]

Hi Aviv,

No, that's not actually the case. It has been a while since I tested these lazy queue scenarios. I re-set my environment, and re-created the test-lazy queue. I then published the following:

1 message of 65536 bytes

1 message of 2048 bytes (lower than the queue_index_embed_msgs_below value of 4096)
65535 messages of 65536 bytes

After restart, RabbitMQ only loads the one message of 2048 bytes into memory.

This document has good explanations - https://www.rabbitmq.com/lazy-queues.html#caveats-limitations

Without knowing the specific details of message size, queue declaration, etc from your environment I can't explain what you're seeing at this point in time.

Thanks -

Luke

[aviv salem]

O.k. Thanks for your (considerable amount of) time....

I'll try to further investigate, and let you know if we encounter the problem again...

Thanks again! 

On Mon, Apr 22, 2019, 20:41 Luke Bakken <lba...@pivotal.io> wrote:

Hi Aviv,

No, that's not actually the case. It has been a while since I tested these lazy queue scenarios. I re-set my environment, and re-created the test-lazy queue. I then published the following:

1 message of 65536 bytes

1 message of 2048 bytes (lower than the queue_index_embed_msgs_below value of 4096)
65535 messages of 65536 bytes

After restart, RabbitMQ only loads the one message of 2048 bytes into memory.

This document has good explanations - https://www.rabbitmq.com/lazy-queues.html#caveats-limitations

Without knowing the specific details of message size, queue declaration, etc from your environment I can't explain what you're seeing at this point in time.

Thanks -

Luke

On Monday, April 22, 2019 at 10:27:52 AM UTC-7, aviv salem wrote:

one small message is enough to load EVERYTHING into the memory?

On Mon, 22 Apr 2019 at 18:29, Luke Bakken <lba...@pivotal.io> wrote:

Hi Aviv,

I can't reproduce what you report. I created a queue named "test-lazy" (durable, lazy) then I published 65536 messages of 65536 bytes each via PerfTest using these arguments:

--predeclared --queue test-lazy --flag persistent --size 65536 --pmessages 65536 --producers 1 --consumers 0

The "test-lazy-65K-msgs.png" file shows the state of that queue after the above command completes.

I then restarted RabbitMQ. The other two files show the state of the queue and overview after the restart. You can see that no messages have been loaded into memory.

In your case, either the queue was not declared lazy prior to publishing those messages, or perhaps there is at least one small message in the data store that triggers loading into memory

--