Added Disk Space Now Publishers Are Blocked

[dru...@certona.com]

Hi,

We have had rabbitmq running on this Linux box for quite some time.   

We have added new disks to the system and presented it as /media/ssd1

We have moved all files from /home/mnesia to /media/ssd1,  set owner, set group, everything matches /home/mnesia

The service starts but says publishers are blocked.  This is because of the messages noted below...

Log -

=INFO REPORT==== 2-Feb-2017::17:02:46 ===

Memory limit set to 735247MB of 773944MB total.

=INFO REPORT==== 2-Feb-2017::17:02:46 ===

Disk free limit set to 81153MB

=INFO REPORT==== 2-Feb-2017::17:02:46 ===

Disk free space insufficient. Free bytes:18360586240 Limit:81153974682

=WARNING REPORT==== 2-Feb-2017::17:02:46 ===

disk resource limit alarm set on node 'rabbit@rab08'.

The /etc/rabbitmq/rabbitmq.config contains a line:

     {disk_free_limit, {mem_relative, 0.1}}

The System has 768GB RAM, the SSDs are 773GB which is more than the RAM.

I work around the issue temporarily by issuing:

rabbitmqctl set_disk_free_limit 1GB

But this obviously does not solve the problem, it just lets the Publishers become unblocked.

How do I get disk space to show more than 17GB on the Web GUI?

Node: rabbit@rab08 (More about this node)

File descriptors	Socket descriptors	Erlang processes	Memory	Disk space	Rates mode	Info	+/-
38 1032768 available	3 929399 available	852 1048576 available	1.6GB 718GB high watermark	17GB 954MB low watermark	basic	Disc 3 Stats

Paths

Config file	/etc/rabbitmq/rabbitmq.config
Database directory	/media/ssdl/mnesia/rabbit@rab08
Log file	/var/log/rabbitmq/rab...@rab08.log
SASL log file	/var/log/rabbitmq/rab...@rab08-sasl.log

[Michael Klishin]

Using absolute value is a better idea, in fact, ` {disk_free_limit, {mem_relative, 0.1}}` is a really poorly thought

out value. Free disk space must be at least x1.1 of the amount of RAM: how do you expect nodes to fit,

say, 50% of RAM it uses into 10% of RAM in free disk space?

RabbitMQ management UI suggests that its database directory is `/media/ssdl/mnesia/rabbit@rab08`,

so if that's the right partition, it's your limit settings that need tweaking (and it looks like they really do).

So bumping the limit using an absolute value is NOT a temporary solution that "just unblocks publishers."

Log file	/var/log/rabbitmq/rabbit@rab08.log
SASL log file	/var/log/rabbitmq/rabbit@rab08-sasl.log

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Dane Ruyle]

Hi,

No matter what I set disk_free_limit to be, RabbitMQ still sees 17GB free.   We need to fix this.

Log file	/var/log/rabbitmq/rab...@rab08.log
SASL log file	/var/log/rabbitmq/rab...@rab08-sasl.log

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

 

--

MK

 

Staff Software Engineer, Pivotal/RabbitMQ

--
You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/yk4f31jP4B4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

[Michael Klishin]

RabbitMQ uses `df -kP /path/to/node/data/directory` on Linux to determine how much disk space is free:

https://github.com/rabbitmq/rabbitmq-server/blob/stable/src/rabbit_disk_monitor.erl#L24

https://github.com/rabbitmq/rabbitmq-server/blob/stable/src/rabbit_disk_monitor.erl#L214

so try running that manually and see what it returns. There isn't any particularly sophisticated

parsing involved.

Node's database directory is displayed in the management UI but you can also see it in `rabbitmqctl environment`

output or by running

rabbitmqctl eval "rabbit_mnesia:dir()."

`rabbit.disk_free_limit` does not control how much disk space is free. It controls the lower watermark

after which RabbitMQ will consider the node to be dangerously out of disk space.

Configuration changes require a node restart. `rabbitmqctl set_disk_free_limit` adjusts the limit

at runtime.

I have missed the part that your node has over 700 GB of RAM. Unless it's typical for your workload to have

100s of GBs of messages in memory, you don't need to have a comparable amount of free disk space.

We have some basic recommendations:

http://www.rabbitmq.com/production-checklist.html

but they cover nodes with smaller amounts of RAM. I'd say for 768 GB you'd be set with 250 or so GB

of free disk space.

Log file	/var/log/rabbitmq/rabbit@rab08.log
SASL log file	/var/log/rabbitmq/rabbit@rab08-sasl.log

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

 

--

MK

 

Staff Software Engineer, Pivotal/RabbitMQ

--
You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/yk4f31jP4B4/unsubscribe.

To unsubscribe from this group and all its topics, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Dane Ruyle]

Thank you, that helps.

Looks like we are using LVM.

LVM has taken all the free space (at the time of install) and made a big volume.

It is not recommended to add spindle and ssd to the same group.

 

I am considering burning the machine down and rebuilding without lvm.

 

 

From: rabbitm...@googlegroups.com [mailto:rabbitm...@googlegroups.com] On Behalf Of Michael Klishin
Sent: Friday, February 03, 2017 12:20 PM
To: rabbitm...@googlegroups.com
Subject: Re: [rabbitmq-users] Added Disk Space Now Publishers Are Blocked

 

RabbitMQ uses `df -kP /path/to/node/data/directory` on Linux to determine how much disk space is free:

Log file	/var/log/rabbitmq/rab...@rab08.log
SASL log file	/var/log/rabbitmq/rab...@rab08-sasl.log

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

 

--

MK

 

Staff Software Engineer, Pivotal/RabbitMQ

--

You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/yk4f31jP4B4/unsubscribe.

To unsubscribe from this group and all its topics, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

 

--

MK

 

Staff Software Engineer, Pivotal/RabbitMQ

--
You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/yk4f31jP4B4/unsubscribe.

To unsubscribe from this group and all its topics, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.