Mnesia DB files folder consuming disk space

[Lulu Paul]

Hi team

We are running RabbitMQ 3.6.0, Erlang 18.2.1 on a Windows 2008 R2 server test server  

The disk space on the C drive is dunning out of space and it shows that RabbitMQ node's Mnesia database files folder is consuming 13 GB space. 

If we cannot increase the disk space of the C drive is it better to move the chnage  the RabbitMQ mnesia database location to the D drive?

I just found this link that tells how to go about this. 

https://myupbeat.wordpress.com/2014/02/07/chaging-rabbitmq-mnesia-database-location/

Please suggest if this would be the recommended workaround?

Thanks

Lulu

[Michael Klishin]

The right thing to do is to consume messages, which is very likely what actually takes disk space.

Please produce a disk usage breakdown of everything under the mnesia directory (which contains way more than Mnesia, the currently used

schema data store) before you do anything else. Don't guess, collect data instead.

Of course you can stop the node, move the directory and point the node to the new location [1] but that won't address the root of the problem.

1. http://www.rabbitmq.com/relocate.html

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Lulu Paul]

Michael,

You are right, thank you for steering  me in the right direction.

The queues folder is the culprit. There are 9 queues of which one is consuming 13 GB. Several .idx files sitting there since 16/11. Not sure what might have caused these to not be consumed. 

Strangely enough, now i am also unable to login to the management console with the registered user ID or the default admin. It says login failed.

How do you reckon we proceed?

 

Thanks,

Lulu

[Michael Klishin]

I cannot comment on the management UI login attempt. See server logs [1] and consider providing the detailed

error message. For example, if you use default administrator credentials then you can only log in from localhost [2].

The files in question are queue index files. They will accumulate when there are no consumers on queues

but a constant stream of routed messages. See your monitoring [3] data to find out whether that might be the case.

if you don't have monitoring in place, we HIGHLY recommend it.

Consumers can also be listed using `rabbitmqctl.bat list_consumers` (note that the command is scoped to just one virtual host).

1. https://www.rabbitmq.com/logging.html

2. https://www.rabbitmq.com/access-control.html#loopback-users

3. http://www.rabbitmq.com/monitoring.html

[Michael Klishin]

Perhaps I should also mention the doc section on management UI access [1].

1. http://www.rabbitmq.com/management.html#usage-ui

[Michael Klishin]

…sections, there's more than one :/

http://www.rabbitmq.com/management.html#permissions

[Michael Klishin]

Last but not least, RabbitMQ 3.6.x has been out of support for about 6 months.

3.6.0 is 16 patch releases behind even in the 3.6.x series [1]. Please consider upgrading [2]

after you address the issue at hand.

Erlang 18.2 is known to have bugs that are catastrophic to RabbitMQ [3].

1. https://www.rabbitmq.com/changelog.html

2. https://www.rabbitmq.com/upgrade.html

3. https://www.rabbitmq.com/which-erlang.html

[Lulu Paul]

Michael,

Thank you. I am able to login with guest account on localhost as you mentioned. Tried to reset the pwd using the change_password command over rabbitmqctl. But the error says Error: unable to connect to node 'rabbit@V8L-APPTEST2': nodedown

Sam error even if I try to perform a rabbitmqctl.bat list_consumers. Is this a hostname resolution issue?

As for the queue index files, This is how it looks now. 

Cheers,

Lulu

[Lulu Paul]

Below is the diagnostics info -

F:\>"C:\Program Files\RabbitMQ Server\rabbitmq_s " list_consumers

Error: unable to connect to node 'rabbit@XXX

DIAGNOSTICS

===========

attempted to contact: ['rabbit@XXXX']

rabbit@XXXX:

  * connected to epmd (port 4369) on XXX

  * epmd reports node 'rabbit' running on port 2

  * TCP connection succeeded but Erlang distribu

  * Authentication failed (rejected by the remote node), please check Erlang cookie

current node details:

- node name: 'rabbitmq-cli-38@XXX'

- home dir: F:\

- cookie hash: XXXXXX====

Thanks,

Lulu

[Michael Klishin]

You have 36 million messages in the IsadgDCPublish queue. See how many consumers you have on that queue

but according to the rates it's likely that you have none. That's the root cause here.

You can purge the queue (delete all of its contents) via management UI. Note that with 36M messages it will take a while and management UI

will claim a timeout.

I don't know if it's a hostname resolution issue, CLI tools even in 3.6.0 try to provide additional context in a diagnostics message.

See [1].

1. http://www.rabbitmq.com/cli.html#erlang-cookie

[Michael Klishin]

You can also delete and redeclare the queue. It will also take a while but might take less time and even

avoid the management UI timeout.

[Michael Klishin]

The node provided all the relevant information in the diagnostics message:

 * TCP connection succeeded but Erlang distribution failed (easily a top 3 question on this list)

 * Authentication failed (rejected by the remote node), please check Erlang cookie

[1] mentions "TCP connection succeeded but Erlang distribution failed" specifically.

1. http://www.rabbitmq.com/cli.html#erlang-cookie