Unable to start RMQ after reclustering & increasing ulimit
Spruha Pattnaik
Status logs:
```
Mar 13 08:22:26 ip-172-31-10-171 systemd[1]: rabbitmq-server.service: Start operation timed out. Terminating. Mar 13 08:22:26 ip-172-31-10-171 rabbitmq-server[20228]: Failed to read from erl_child_setup: 104 Mar 13 08:22:26 ip-172-31-10-171 rabbitmq-server[20228]: [1B blob data] Mar 13 08:22:26 ip-172-31-10-171 systemd[1]: rabbitmq-server.service: Main process exited, code=exited, status=1/FAILURE Mar 13 08:22:26 ip-172-31-10-171 systemd[1]: Failed to start RabbitMQ broker.
```
RMQ logs:
```
2023-03-13 07:24:18.757 [info] <0.33.0> Application mnesia started on node 'rabbit@ip-172-31-10-171'
2023-03-13 07:24:18.757 [info] <0.33.0> Application crypto started on node 'rabbit@ip-172-31-10-171'
2023-03-13 07:24:18.758 [info] <0.33.0> Application cowlib started on node 'rabbit@ip-172-31-10-171'
2023-03-13 07:24:18.758 [info] <0.33.0> Application xmerl started on node 'rabbit@ip-172-31-10-171'
2023-03-13 07:24:18.758 [info] <0.33.0> Application asn1 started on node 'rabbit@ip-172-31-10-171'
2023-03-13 07:24:18.758 [info] <0.33.0> Application public_key started on node 'rabbit@ip-172-31-10-171'
2023-03-13 07:24:18.758 [info] <0.33.0> Application ssl started on node 'rabbit@ip-172-31-10-171'
2023-03-13 07:24:18.758 [info] <0.33.0> Application ranch started on node 'rabbit@ip-172-31-10-171'
2023-03-13 07:24:18.762 [info] <0.33.0> Application cowboy started on node 'rabbit@ip-172-31-10-171'
2023-03-13 07:24:18.762 [info] <0.33.0> Application ranch_proxy_protocol started on node 'rabbit@ip-172-31-10-171'
2023-03-13 07:24:18.763 [info] <0.33.0> Application rabbit_common started on node 'rabbit@ip-172-31-10-171'
2023-03-13 07:24:18.765 [info] <0.33.0> Application amqp_client started on node 'rabbit@ip-172-31-10-171'
2023-03-13 07:24:18.766 [info] <0.255.0>
Starting RabbitMQ 3.7.4 on Erlang 20.3
Copyright (C) 2007-2018 Pivotal Software, Inc.
Licensed under the MPL. See http://www.rabbitmq.com/
2023-03-13 07:24:18.766 [info] <0.255.0>
node : rabbit@ip-172-31-10-171
home dir : /var/lib/rabbitmq
config file(s) : /etc/rabbitmq/rabbitmq.conf
cookie hash : a6G3u2b1cQh/8JbQ4Sdobg==
log(s) : /data/logs/rab...@ip-172-31-10-171.log
: /data/logs/rabbit@ip-172-31-10-171_upgrade.log
database dir : /data/rabbitmq/rabbit@ip-172-31-10-171
2023-03-13 07:24:19.124 [info] <0.288.0> Memory high watermark set to 9536 MiB (10000000000 bytes) of 15669 MiB (16430768128 bytes) total
2023-03-13 07:24:19.133 [info] <0.290.0> Enabling free disk space monitoring
2023-03-13 07:24:19.133 [info] <0.290.0> Disk free limit set to 50MB
2023-03-13 07:24:19.139 [info] <0.292.0> Limiting to approx 65435 file handles (58889 sockets)
2023-03-13 07:24:19.139 [info] <0.293.0> FHC read buffering: OFF
2023-03-13 07:24:19.139 [info] <0.293.0> FHC write buffering: ON
2023-03-13 08:53:28.849 [info] <0.33.0> Application lager started on node 'rabbit@ip-172-31-10-171'
2023-03-13 08:53:28.854 [info] <0.5.0> Log file opened with Lager
2023-03-13 08:53:31.357 [info] <0.33.0> Application mnesia started on node 'rabbit@ip-172-31-10-171'
2023-03-13 08:53:31.359 [info] <0.33.0> Application mnesia exited with reason: stopped
2023-03-13 08:53:32.119 [info] <0.33.0> Application inets started on node 'rabbit@ip-172-31-10-171'
2023-03-13 08:53:32.119 [info] <0.33.0> Application os_mon started on node 'rabbit@ip-172-31-10-171'
2023-03-13 08:53:32.119 [info] <0.33.0> Application amqp10_common started on node 'rabbit@ip-172-31-10-171'
2023-03-13 08:53:32.119 [info] <0.33.0> Application jsx started on node 'rabbit@ip-172-31-10-171'
2023-03-13 08:53:32.119 [info] <0.33.0> Application recon started on node 'rabbit@ip-172-31-10-171'
2023-03-13 08:53:32.136 [error] <0.178.0> Mnesia('rabbit@ip-172-31-10-171'): ** ERROR ** mnesia_event got {inconsistent_database, starting_partitioned_network, 'rabbit@ip-172-31-0-242'}
```