stream state down after node reboot

[Yohann Vanfrachem]

Hello, I use Rabbit 3.11.3 and Erlang 25.1.2 on w10 

I have a 3 nodes cluster with cluster_partition_handling = pause_minority 

A stream_queue is replicated on all cluster nodes with state: running

I don't know why but after a reboot of machine W10-DINF-04 I got state "down" for this stream on another machine

Error log from the machine I rebooted with the state "down" :

2022-11-29 15:39:02.923000+01:00 [info] <0.1629.0> node 'rabbit@W10-DINF-04' up
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>   crasher:
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>     initial call: cowboy_stream_h:request_process/3
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>     pid: <0.19000.1>
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>     registered_name: []
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>     exception error: bad argument
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>       in operator  ++/2
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>          called as {badrpc,
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>                     {'EXIT',
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>                      {badarg,
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>                       [{ets,match_object,
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>                         [rabbit_listener_ets,'_'],
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>                         [{error_info,
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>                           #{cause => id,module => erl_stdlib_errors}}]},
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>                        {ets,tab2list,1,[{file,"ets.erl"},{line,771}]}]}}}
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>                    ++
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>                    [{listener,'rabbit@w10-valid-ps',http,"w10-valid-ps",
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>                               {0,0,0,0,0,0,0,0},
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>                               15672,
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>                               [{cowboy_opts,[{sendfile,false}]},{port,15672}]},
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>                     {listener,'rabbit@w10-valid-ps',clustering,"w10-valid-ps",
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>                               {0,0,0,0,0,0,0,0},
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>                               25672,[]},
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>                     {listener,'rabbit@w10-valid-ps',amqp,"w10-valid-ps",
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>                               {0,0,0,0,0,0,0,0},
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>                               5672,
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>                               [{backlog,128},
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>                                {nodelay,true},
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>                                {linger,{true,0}},
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>                                {exit_on_close,false}]},
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>                     {listener,'rabbit@w10-valid-ps',amqp,"w10-valid-ps",
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>                               {0,0,0,0},
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>                               5672,
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>                               [{backlog,128},
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>                                {nodelay,true},
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>                                {linger,{true,0}},
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>                                {exit_on_close,false}]},
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>                     {listener,'rabbit@W10-DINF-02',http,"W10-DINF-02",
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>                               {0,0,0,0,0,0,0,0},
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>                               15672,
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>                               [{cowboy_opts,[{sendfile,false}]},{port,15672}]},
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>                     {listener,'rabbit@W10-DINF-02',clustering,"W10-DINF-02",
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>                               {0,0,0,0,0,0,0,0},
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>                               25672,[]},
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>                     {listener,'rabbit@W10-DINF-02',amqp,"W10-DINF-02",
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>                               {0,0,0,0,0,0,0,0},
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>                               5672,
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>                               [{backlog,128},
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>                                {nodelay,true},
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>                                {linger,{true,0}},
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>                                {exit_on_close,false}]},
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>                     {listener,'rabbit@W10-DINF-02',amqp,"W10-DINF-02",
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>                               {0,0,0,0},
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>                               5672,
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>                               [{backlog,128},
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>                                {nodelay,true},
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>                                {linger,{true,0}},
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>                                {exit_on_close,false}]}]
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>       in call from lists:append/1 (lists.erl, line 155)
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>       in call from rabbit_mgmt_wm_overview:listeners/0 (rabbit_mgmt_wm_overview.erl, line 111)
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>       in call from rabbit_mgmt_wm_overview:to_json/2 (rabbit_mgmt_wm_overview.erl, line 66)
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>       in call from cowboy_rest:call/3 (src/cowboy_rest.erl, line 1575)
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>       in call from cowboy_rest:set_resp_body/2 (src/cowboy_rest.erl, line 1464)
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>       in call from cowboy_rest:upgrade/4 (src/cowboy_rest.erl, line 284)
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>       in call from cowboy_stream_h:execute/3 (src/cowboy_stream_h.erl, line 300)
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>     ancestors: [<0.16917.1>,<0.1803.0>,<0.1797.0>,<0.1796.0>,<0.1794.0>,
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>                   rabbit_web_dispatch_sup,<0.1783.0>]
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>     message_queue_len: 0
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>     messages: []
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>     links: [<0.16917.1>]
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>     dictionary: []
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>     trap_exit: false
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>     status: running
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>     heap_size: 4185
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>     stack_size: 28
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>     reductions: 4179
2022-11-29 15:39:03.766000+01:00 [error] <0.19000.1>   neighbours:

Another reboot fixed the problem completely be we would like this to not happen again.

If you have any hints ?

Thanks !