Nodes fail randomly in Rabbitmq cluster

[Mohammed Reehan]

Hello All,

I am using Rabbitmq 3 Node cluster with version 3.7.7. and erlang 22.0.1

off lately i have seen rabbitmq Node dies without any error and when i check on the server, the service is running. 

but i see below errors and i am currently clue less as to what could be the error message to check in /var/log/rabbitmq.log

below are some screenshot. 

019-07-16 09:23:32.842 [error] <0.11002.1> Supervisor rabbit_core_metrics_gc_sup had child rabbit_core_metrics_gc started with rabbit_core_metrics_gc:start_link() at <0.29744.363> exit with reason {aborted,{no_exists,[rabbit_queue,{amqqueue,'_','_','_','_','_','_','_','_','_','_','_','_','_','_','_','_','_','_'}]}} in context child_terminated
2019-07-16 09:23:34.956 [error] <0.30461.363> CRASH REPORT Process <0.30461.363> with 0 neighbours exited with reason: {aborted,{no_exists,[rabbit_runtime_parameters,cluster_name]}} in mnesia:abort/1 line 355
2019-07-16 09:23:34.957 [error] <0.30456.363> Supervisor {<0.30456.363>,rabbit_connection_sup} had child reader started with rabbit_reader:start_link(<0.30453.363>, {acceptor,{0,0,0,0,0,0,0,0},5672}, #Port<0.662291>) at <0.30461.363> exit with reason {aborted,{no_exists,[rabbit_runtime_parameters,cluster_name]}} in context child_terminated
2019-07-16 09:23:34.957 [error] <0.30456.363> Supervisor {<0.30456.363>,rabbit_connection_sup} had child reader started with rabbit_reader:start_link(<0.30453.363>, {acceptor,{0,0,0,0,0,0,0,0},5672}, #Port<0.662291>) at <0.30461.363> exit with reason reached_max_restart_intensity in context shutdown
2019-07-16 09:23:35.143 [error] <0.30459.363> CRASH REPORT Process <0.30459.363> with 0 neighbours exited with reason: {aborted,{no_exists,[rabbit_runtime_parameters,cluster_name]}} in mnesia:abort/1 line 355
2019-07-16 09:23:35.143 [error] <0.30462.363> Supervisor {<0.30462.363>,rabbit_connection_sup} had child reader started with rabbit_reader:start_link(<0.30460.363>, {acceptor,{0,0,0,0,0,0,0,0},5672}, #Port<0.662278>) at <0.30459.363> exit with reason {aborted,{no_exists,[rabbit_runtime_parameters,cluster_name]}} in context child_terminated
2019-07-16 09:23:35.144 [error] <0.30462.363> Supervisor {<0.30462.363>,rabbit_connection_sup} had child reader started with rabbit_reader:start_link(<0.30460.363>, {acceptor,{0,0,0,0,0,0,0,0},5672}, #Port<0.662278>) at <0.30459.363> exit with reason reached_max_restart_intensity in context shutdown
2019-07-16 09:23:36.151 [error] <0.30470.363> CRASH REPORT Process <0.30470.363> with 0 neighbours exited with reason: {aborted,{no_exists,[rabbit_runtime_parameters,cluster_name]}} in mnesia:abort/1 line 355
2019-07-16 09:23:36.152 [error] <0.30463.363> Supervisor {<0.30463.363>,rabbit_connection_sup} had child reader started with rabbit_reader:start_link(<0.30464.363>, {acceptor,{0,0,0,0,0,0,0,0},5672}, #Port<0.662301>) at <0.30470.363> exit with reason {aborted,{no_exists,[rabbit_runtime_parameters,cluster_name]}} in context child_terminated
2019-07-16 09:23:36.153 [error] <0.30463.363> Supervisor {<0.30463.363>,rabbit_connection_sup} had child reader started with rabbit_reader:start_link(<0.30464.363>, {acceptor,{0,0,0,0,0,0,0,0},5672}, #Port<0.662301>) at <0.30470.363> exit with reason reached_max_restart_intensity in context shutdown
2019-07-16 09:23:37.557 [error] <0.30476.363> CRASH REPORT Process <0.30476.363> with 0 neighbours exited with reason: {aborted,{no_exists,[rabbit_runtime_parameters,cluster_name]}} in mnesia:abort/1 line 355
2019-07-16 09:23:37.558 [error] <0.30465.363> Supervisor {<0.30465.363>,rabbit_connection_sup} had child reader started with rabbit_reader:start_link(<0.30475.363>, {acceptor,{0,0,0,0,0,0,0,0},5672}, #Port<0.662297>) at <0.30476.363> exit with reason {aborted,{no_exists,[rabbit_runtime_parameters,cluster_name]}} in context child_terminated
2019-07-16 09:23:37.558 [error] <0.30465.363> Supervisor {<0.30465.363>,rabbit_connection_sup} had child reader started with rabbit_reader:start_link(<0.30475.363>, {acceptor,{0,0,0,0,0,0,0,0},5672}, #Port<0.662297>) at <0.30476.363> exit with reason reached_max_restart_intensity in context shutdown
2019-07-16 09:23:41.928 [error] <0.30480.363> CRASH REPORT Process <0.30480.363> with 0 neighbours exited with reason: {aborted,{no_exists,[rabbit_runtime_parameters,cluster_name]}} in mnesia:abort/1 line 355
2019-07-16 09:23:41.929 [error] <0.30477.363> Supervisor {<0.30477.363>,rabbit_connection_sup} had child reader started with rabbit_reader:start_link(<0.30478.363>, {acceptor,{0,0,0,0,0,0,0,0},5672}, #Port<0.662308>) at <0.30480.363> exit with reason {aborted,{no_exists,[rabbit_runtime_parameters,cluster_name]}} in context child_terminated
2019-07-16 09:23:41.929 [error] <0.30477.363> Supervisor {<0.30477.363>,rabbit_connection_sup} had child reader started with rabbit_reader:start_link(<0.30478.363>, {acceptor,{0,0,0,0,0,0,0,0},5672}, #Port<0.662308>) at <0.30480.363> exit with reason reached_max_restart_intensity in context shutdown
2019-07-16 09:23:42.641 [error] <0.30482.363> CRASH REPORT Process <0.30482.363> with 0 neighbours exited with reason: {aborted,{no_exists,[rabbit_runtime_parameters,cluster_name]}} in mnesia:abort/1 line 355
2019-07-16 09:23:42.642 [error] <0.30471.363> Supervisor {<0.30471.363>,rabbit_connection_sup} had child reader started with rabbit_reader:start_link(<0.30481.363>, {acceptor,{0,0,0,0,0,0,0,0},5672}, #Port<0.662311>) at <0.30482.363> exit with reason {aborted,{no_exists,[rabbit_runtime_parameters,cluster_name]}} in context child_terminated
2019-07-16 09:23:42.642 [error] <0.30471.363> Supervisor {<0.30471.363>,rabbit_connection_sup} had child reader started with rabbit_reader:start_link(<0.30481.363>, {acceptor,{0,0,0,0,0,0,0,0},5672}, #Port<0.662311>) at <0.30482.363> exit with reason reached_max_restart_intensity in context shutdown
2019-07-16 09:23:46.202 [error] <0.17578.363> CRASH REPORT Process <0.17578.363> with 0 neighbours exited with reason: {aborted,{no_exists,[rabbit_runtime_parameters,cluster_name]}} in mnesia:abort/1 line 355
2019-07-16 09:23:46.203 [error] <0.30487.363> Supervisor {<0.30487.363>,rabbit_connection_sup} had child reader started with rabbit_reader:start_link(<0.30486.363>, {acceptor,{0,0,0,0,0,0,0,0},5672}, #Port<0.662318>) at <0.17578.363> exit with reason {aborted,{no_exists,[rabbit_runtime_parameters,cluster_name]}} in context child_terminated
2019-07-16 09:23:46.203 [error] <0.30487.363> Supervisor {<0.30487.363>,rabbit_connection_sup} had child reader started with rabbit_reader:start_link(<0.30486.363>, {acceptor,{0,0,0,0,0,0,0,0},5672}, #Port<0.662318>) at <0.17578.363> exit with reason reached_max_restart_intensity in context shutdown
2019-07-16 09:23:49.023 [error] <0.30692.363> CRASH REPORT Process <0.30692.363> with 0 neighbours exited with reason: {aborted,{no_exists,[rabbit_runtime_parameters,cluster_name]}} in mnesia:abort/1 line 355
2019-07-16 09:23:49.024 [error] <0.30688.363> Supervisor {<0.30688.363>,rabbit_connection_sup} had child reader started with rabbit_reader:start_link(<0.30695.363>, {acceptor,{0,0,0,0,0,0,0,0},5672}, #Port<0.662303>) at <0.30692.363> exit with reason {aborted,{no_exists,[rabbit_runtime_parameters,cluster_name]}} in context child_terminated
2019-07-16 09:23:49.024 [error] <0.30688.363> Supervisor {<0.30688.363>,rabbit_connection_sup} had child reader started with rabbit_reader:start_link(<0.30695.363>, {acceptor,{0,0,0,0,0,0,0,0},5672}, #Port<0.662303>) at <0.30692.363> exit with reason reached_max_restart_intensity in context shutdown
2019-07-16 09:23:49.372 [error] <0.30701.363> CRASH REPORT Process <0.30701.363> with 0 neighbours exited with reason: {aborted,{no_exists,[rabbit_runtime_parameters,cluster_name]}} in mnesia:abort/1 line 355
2019-07-16 09:23:49.372 [error] <0.30691.363> Supervisor {<0.30691.363>,rabbit_connection_sup} had child reader started with rabbit_reader:start_link(<0.30694.363>, {acceptor,{0,0,0,0,0,0,0,0},5672}, #Port<0.662325>) at <0.30701.363> exit with reason {aborted,{no_exists,[rabbit_runtime_parameters,cluster_name]}} in context child_terminated
2019-07-16 09:23:49.373 [error] <0.30691.363> Supervisor {<0.30691.363>,rabbit_connection_sup} had child reader started with rabbit_reader:start_link(<0.30694.363>, {acceptor,{0,0,0,0,0,0,0,0},5672}, #Port<0.662325>) at <0.30701.363> exit with reason reached_max_restart_intensity in context shutdown
2019-07-16 09:23:51.606 [error] <0.30698.363> CRASH REPORT Process <0.30698.363> with 0 neighbours exited with reason: {aborted,{no_exists,[rabbit_runtime_parameters,cluster_name]}} in mnesia:abort/1 line 355
2019-07-16 09:23:51.607 [error] <0.30690.363> Supervisor {<0.30690.363>,rabbit_connection_sup} had child reader started with rabbit_reader:start_link(<0.30699.363>, {acceptor,{0,0,0,0,0,0,0,0},5672}, #Port<0.662320>) at <0.30698.363> exit with reason {aborted,{no_exists,[rabbit_runtime_parameters,cluster_name]}} in context child_terminated
2019-07-16 09:23:51.607 [error] <0.30690.363> Supervisor {<0.30690.363>,rabbit_connection_sup} had child reader started with rabbit_reader:start_link(<0.30699.363>, {acceptor,{0,0,0,0,0,0,0,0},5672}, #Port<0.662320>) at <0.30698.363> exit with reason reached_max_restart_intensity in context shutdown
2019-07-16 09:23:52.417 [error] <0.30700.363> CRASH REPORT Process <0.30700.363> with 0 neighbours exited with reason: {aborted,{no_exists,[rabbit_runtime_parameters,cluster_name]}} in mnesia:abort/1 line 355
2019-07-16 09:23:52.418 [error] <0.30706.363> Supervisor {<0.30706.363>,rabbit_connection_sup} had child reader started with rabbit_reader:start_link(<0.30704.363>, {acceptor,{0,0,0,0,0,0,0,0},5672}, #Port<0.662329>) at <0.30700.363> exit with reason {aborted,{no_exists,[rabbit_runtime_parameters,cluster_name]}} in context child_terminated
2019-07-16 09:23:52.419 [error] <0.30706.363> Supervisor {<0.30706.363>,rabbit_connection_sup} had child reader started with rabbit_reader:start_link(<0.30704.363>, {acceptor,{0,0,0,0,0,0,0,0},5672}, #Port<0.662329>) at <0.30700.363> exit with reason reached_max_restart_intensity in context shutdown
2019-07-16 09:23:55.168 [error] <0.13474.363> CRASH REPORT Process <0.13474.363> with 0 neighbours exited with reason: {aborted,{no_exists,[rabbit_runtime_parameters,cluster_name]}} in mnesia:abort/1 line 355
2019-07-16 09:23:55.169 [error] <0.30703.363> Supervisor {<0.30703.363>,rabbit_connection_sup} had child reader started with rabbit_reader:start_link(<0.30702.363>, {acceptor,{0,0,0,0,0,0,0,0},5672}, #Port<0.662326>) at <0.13474.363> exit with reason {aborted,{no_exists,[rabbit_runtime_parameters,cluster_name]}} in context child_terminated
2019-07-16 09:23:55.170 [error] <0.30703.363> Supervisor {<0.30703.363>,rabbit_connection_sup} had child reader started with rabbit_reader:start_link(<0.30702.363>, {acceptor,{0,0,0,0,0,0,0,0},5672}, #Port<0.662326>) at <0.13474.363> exit with reason reached_max_restart_intensity in context shutdown
2019-07-16 09:23:57.282 [error] <0.13441.363> CRASH REPORT Process <0.13441.363> with 0 neighbours exited with reason: {aborted,{no_exists,[rabbit_runtime_parameters,cluster_name]}} in mnesia:abort/1 line 355
2019-07-16 09:23:57.283 [error] <0.13473.363> Supervisor {<0.13473.363>,rabbit_connection_sup} had child reader started with rabbit_reader:start_link(<0.13524.363>, {acceptor,{0,0,0,0,0,0,0,0},5672}, #Port<0.662330>) at <0.13441.363> exit with reason {aborted,{no_exists,[rabbit_runtime_parameters,cluster_name]}} in context child_terminated
2019-07-16 09:23:57.283 [error] <0.13473.363> Supervisor {<0.13473.363>,rabbit_connection_sup} had child reader started with rabbit_reader:start_link(<0.13524.363>, {acceptor,{0,0,0,0,0,0,0,0},5672}, #Port<0.662330>) at <0.13441.363> exit with reason reached_max_restart_intensity in context shutdown
2019-07-16 09:23:59.609 [error] <0.13372.363> CRASH REPORT Process <0.13372.363> with 0 neighbours exited with reason: {aborted,{no_exists,[rabbit_runtime_parameters,cluster_name]}} in mnesia:abort/1 line 355

--

Mohammed Rehan

[Mohammed Reehan]

Hello Team,

Please help me to fix this, any suggestions

--

Mohammed Rehan

[Michael Klishin]

According to this message an internal table used to store runtime parameters does not exist.

Please start by moving to 3.7.17.

On Wednesday, July 17, 2019 at 5:06:26 PM UTC+3, Mohammed Reehan wrote:

Hello Team,

Please help me to fix this, any suggestions