Unable to recover vhost <<"/">> error and Rabbitmq does not startup as expected

[Arun Kumar]

I have installed RabbitMQ 3.7.16 on Erlang 21.3 on a single Windows 10 system and created two Exchanges A and B, multiple Queues and also shovels for messages to flow between Queues of Exchange A to Exchange B and vice versa. The system was working fine as expected for few days. Due to some issues in my system (Issue: System ran out of memory due to some other process running on that system), I restarted the system after which the rabbitmq is not working as expected. Rabbitmq starts but the Exchanges, Queues, Connections are missing and all shovels are in starting state. Upon restarting the rabbitmq separately and checking the logs, I find that the vhost "/" has not been starting and I do not understand the issue. I checked online for some solutions but I could not find any. Attached the rabbitmq startup log file for your reference.

1. What is the issue that vhost "/" is not starting up?

2. Is vhost "/" corrupted? If yes, what can be the possible reasons for vhost corruption?

3. Can we recover my default vhost "/" without loosing my rabbitmq data such as exchanges, queues etc?

Please provide help.

[Wesley Peng]

Arun Kumar wrote:
> 1. What is the issue that vhost "/" is not starting up?
> 2. Is vhost "/" corrupted? If yes, what can be the possible reasons for
> vhost corruption?
> 3. Can we recover my default vhost "/" without loosing my rabbitmq data
> such as exchanges, queues etc?
>

1 and 2, means rabbitmq not started correctly.
The OS (Windows) has a file handler limit, not sure what the limit is,
but when reached, an error as yours is thrown.

if you want no data losing during rabbitmq restart, try to declare
exchange, queues and messages with persistent mode.

regards.

[Arun Kumar]

Thanks Wesley. I restarted the entire system and yet rabbitmq gives the same error on startup even though all other processes are down. Also we have been using persistent Exchanges and queues. The messages are also declared to be persistent (Even when our application is shutdown, rabbitmq exchange, queues, messages are persistent and was seen on rabbitmq management UI before). 

Now with current state of the machine and now that this error has occured, can we somehow recover our rabbitmq ?

[Wesley Peng]

Have you checked system logs especially the filesystem?

regards

on 2019/10/30 13:48, Arun Kumar wrote:
> Thanks Wesley*. *I restarted the entire system and yet rabbitmq gives

[Arun Kumar]

Yes I checked the Windows system logs and there are no issues after startup.After restarting , I checked the system logs and ErlSrv : RabbitMQ: Erlang service started successfully is the message. No other errors no matter how many times i restart the system. The system memory, file handles are alright now, But rabbitmq logs has the same errors on startup as above and the Vhost"/" is not recovered. Rabbitmq management UI shows the system file handles, memory etc. I have attached the rabbitmq management UI snapshot for your reference. Let me know if you need me to share anything else.

[Wesley Peng]

Arun Kumar wrote:
> Yes I checked the Windows system logs and there are no issues after
> startup.After restarting , I checked the system logs and ErlSrv
> : RabbitMQ: Erlang service started successfully is the message. No other
> errors no matter how many times i restart the system. The system memory,
> file handles are alright now, But rabbitmq logs has the same errors on
> startup as above and the Vhost"/" is not recovered. Rabbitmq management
> UI shows the system file handles, memory etc. I have attached the
> rabbitmq management UI snapshot for your reference. Let me know if you
> need me to share anything else.

Hello

Can you reset this node to see what happens?

rabbitmqctl stop_app
rabbitmqctl reset
rabbitmqctl start_app

https://www.rabbitmq.com/rabbitmqctl.8.html

regards.

[Arun Kumar]

Hi Wesley,

I tried your solution to reset rabbitmq and it wiped out all the rabbitmq data. We had set of rabbitmq users, some queues created manually etc and all of this vanished after reset. I have attached the rabbitmq log file after reset. Please check and let me know if it can be recovered or not at all.

[Wesley Peng]

Hello

on 2019/11/5 15:15, Arun Kumar wrote:
> I tried your solution to reset rabbitmq and it wiped out all the
> rabbitmq data. We had set of rabbitmq users, some queues created
> manually etc and all of this vanished after reset. I have attached the
> rabbitmq log file after reset. Please check and let me know if it can be
> recovered or not at all.
>

I saw rabbitmq service started normally, no error happened.
So the next you would be able to add metadata (vhost, exchange, queue
etc) by hand or by scripts.
No, it can't get auto-recoverd for metadata from last reset.

regards.

[Arun Kumar]

So we can never recover from such  vhost "/"  not starting up error? How can we prevent such error scenarios if it happens in production environement? Or rather, we do not want to loose out our rabbitmq data in production environment and if vhost "/"  not starting up error occurs again, how do we reinstate our customer data?

[Wesley Peng]

Hi

on 2019/11/6 17:02, Arun Kumar wrote:
> So we can never recover from such  vhost "/"  not starting up error? How
> can we prevent such error scenarios if it happens in production
> environement? Or rather, we do not want to loose out our rabbitmq data
> in production environment and if vhost "/"  not starting up error occurs
> again, how do we reinstate our customer data?

Firstly please confirm me, after reset if the old issue (vhost down) got
resolved?

regards.

[Luke Bakken]

Hi Arun,

The "reset" command does wipe out all data. It fixed the issue with the default ("/") vhost in your environment.

I believe you could have removed the message store files for that vhost to fix this issue. You would have lost your messages, but you would not have had to reset the node.

As you saw, a full disk can be a catastrophic event. You must provision enough space, and monitor your system.

Thanks,

Luke

[Stuart Johnston]

To stop a full disk you can set the disk_free_limit 

Once rabbitMQ reaches the set limit it will not accept any publisher requests until disk limit is below the threshold

It doesn't block any consumers.

In production we have this set to 50MB 

cat /etc/rabbitmq/rabbitmq.conf

disk_free_limit.absolute = 524MB

But as the others have said you have to provision enough disk space 

[Wesley Peng]

on 2019/11/7 1:00, Stuart Johnston wrote:
> To stop a full disk you can set the disk_free_limit
>
> Once rabbitMQ reaches the set limit it will not accept any publisher
> requests until disk limit is below the threshold
> It doesn't block any consumers.
>
> In production we have this set to 50MB
>
> cat /etc/rabbitmq/rabbitmq.conf
>
> disk_free_limit.absolute = 524MB

That's good experience. thank you.
Is there a memory limit also?

regards.

[Arun Kumar]

Hi Luke,

Thanks for pitching in to this conversation. 

How to remove the message store files for that vhost? Should I delete specific directories or the entire contents under - "RabbitMQ\db\rabbit@Sys_INKHOSTWSCT_1571139492423-mnesia\msg_stores\vhosts\628WB79CIFDYO9LJI6DKMI09L ( I see there are "msg_store_persistent", "msg_store_transient", "queues" directories and .vhost, recovery.dets files )

As Johnston mentioned we have set  disk_free_limit = 8GB.

[Arun Kumar]

Hi,

Answering my own question taking Luke's advice so that this might help others -

I deleted the directory under msg_stores\vhosts\ and restarted the rabbitmq. Rabbitmq and vhost "/" restarted fine with all my old exchanges, queues and shovels.