After Rolling restart of a 3 node cluster, Queues defined as SAC have no consumers

[appscurry]

RabbitMQ version 3.8.11

We have a 3 node rabbitMQ cluster, with multiple queues. Some of these queues are defined as a Single Active Consumer(SAC). After rolling restart of the nodes within cluster, the queues which are defined as SAC have NO CONSUMERS and others are fine. 

All Queue Types are mirrored and HA.

ha-mode:exactly

ha-params:3

ha-sync-mode:automatic

Exception in logs: 

channel error; protocol method: #method<channel.close>(reply-code=404, reply-text=NOT_FOUND - queue 'single-active-consumer-queue' in vhost '/dev' has crashed and failed to restart, class-id=50, method-id=10)

channel error; protocol method: #method<channel.close>(reply-code=404, reply-text=NOT_FOUND - failed to perform operation on queue 'single-active-consumer-queue' in vhost '/dev' due to timeout, class-id=50, method-id=10)

[appscurry]

Exception in the Rabbitmq server logs:

2021-03-21 18:40:10.875 [error] <0.1539.0> ** Generic server <0.1539.0> terminating

** Last message in was {'$gen_cast',{force_event_refresh,#Ref<17362.3488093943.2208038913.107208>}}

** When Server state == {q,{amqqueue,{resource,<<"/dev">>,queue,<<"Test-My-Lovely-SAC">>},true,false,none,[{<<"x-single-active-consumer">>,bool,true}],<0.1539.0>,[<17362.1328.0>,<17363.1061.0>],[<17362.1328.0>,<17363.1061.0>],[rabbit@rabbitmq2,rabbit@rabbitmq3],[{vhost,<<"/dev">>},{name,<<"ha-mode">>},{pattern,<<".*">>},{'apply-to',<<"all">>},{definition,[{<<"ha-mode">>,<<"exactly">>},{<<"ha-params">>,3},{<<"ha-sync-mode">>,<<"automatic">>}]},{priority,0}],undefined,[{<17363.1062.0>,<17363.1061.0>},{<17362.1329.0>,<17362.1328.0>},{<0.1542.0>,<0.1539.0>}],[],live,0,[],<<"/dev">>,#{user => <<"infra">>},rabbit_classic_queue,#{}},{<0.1535.0>,{consumer,<<"amq.ctag-inTYKurUD8OQ1NAJyjK_Jw">>,true,250,[],<<"infra">>}},true,rabbit_mirror_queue_master,{state,{resource,<<"/dev">>,queue,<<"Test-My-Lovely-SAC">>},<0.1542.0>,<0.1541.0>,rabbit_priority_queue,{passthrough,rabbit_variable_queue,{vqstate,{0,{[],[]}},{0,{[],[]}},{delta,undefined,0,0,undefined},{0,{[],[]}},{0,{[],[]}},0,{0,nil},{0,nil},{0,nil},{qistate,"/var/lib/rabbitmq/mnesia/rabbit@rabbitmq1/msg_stores/vhosts/5CHD4UJCVKIFW3MTX21AUXPKJ/queues/1GTCEFD33P87OXME2VLRFT4G5",{#{},[]},undefined,0,32768,#Fun<rabbit_variable_queue.2.117506867>,#Fun<rabbit_variable_queue.3.117506867>,{0,nil},{0,nil},[],[],{resource,<<"/dev">>,queue,<<"Test-My-Lovely-SAC">>}},{{client_msstate,<0.703.0>,<<150,219,91,140,226,180,93,64,126,78,128,39,60,34,207,216>>,#{},{state,#Ref<0.1110475327.2208169986.12656>,"/var/lib/rabbitmq/mnesia/rabbit@rabbitmq1/msg_stores/vhosts/5CHD4UJCVKIFW3MTX21AUXPKJ/msg_store_persistent"},rabbit_msg_store_ets_index,"/var/lib/rabbitmq/mnesia/rabbit@rabbitmq1/msg_stores/vhosts/5CHD4UJCVKIFW3MTX21AUXPKJ/msg_store_persistent",<0.706.0>,#Ref<0.1110475327.2208169986.12657>,#Ref<0.1110475327.2208169986.12652>,#Ref<0.1110475327.2208169986.12658>,#Ref<0.1110475327.2208169986.12659>,{4000,800}},{client_msstate,<0.699.0>,<<158,57,200,27,2,195,...>>,...}},...}},...},...}

** Reason for termination == 

** {{badmatch,[{<0.1535.0>,<<"amq.ctag-inTYKurUD8OQ1NAJyjK_Jw">>,true,250,true,up,[],<<"infra">>}]},[{rabbit_amqqueue_process,handle_cast,2,[{file,"src/rabbit_amqqueue_process.erl"},{line,1637}]},{gen_server2,handle_msg,2,[{file,"src/gen_server2.erl"},{line,1067}]},{proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,236}]}]}

2021-03-21 18:40:10.875 [info] <0.1539.0> [{initial_call,{rabbit_prequeue,init,['Argument__1']}},{pid,<0.1539.0>},{registered_name,[]},{error_info,{exit,{{badmatch,[{<0.1535.0>,<<"amq.ctag-inTYKurUD8OQ1NAJyjK_Jw">>,true,250,true,up,[],<<"infra">>}]},[{rabbit_amqqueue_process,handle_cast,2,[{file,"src/rabbit_amqqueue_process.erl"},{line,1637}]},{gen_server2,handle_msg,2,[{file,"src/gen_server2.erl"},{line,1067}]},{proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,236}]}]},[{gen_server2,terminate,3,[{file,"src/gen_server2.erl"},{line,1183}]},{proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,236}]}]}},{ancestors,[<0.1538.0>,<0.710.0>,<0.694.0>,<0.693.0>,rabbit_vhost_sup_sup,rabbit_sup,<0.272.0>]},{message_queue_len,0},{messages,[]},{links,[<0.1541.0>,<0.1538.0>]},{dictionary,[{rand_seed,{#{jump => #Fun<rand.3.47293030>,max => 288230376151711743,next => #Fun<rand.5.47293030>,type => exsplus},[223235698448823691|222974903414534747]}},{guid,{{2654586907,46339452,1085233795,2089653595},1}},{process_name,{rabbit_amqqueue_process,{resource,<<"/dev">>,queue,<<"Test-My-Lovely-SAC">>}}},{{ch,<0.1535.0>},{cr,<0.1535.0>,#Ref<0.1110475327.2208038914.20357>,{0,{[],[]}},1,{queue,[],[],0},{qstate,<0.1534.0>,dormant,{1,{<<"amq.ctag-inTYKurUD8OQ1NAJyjK_Jw">>,{credit,250,auto},nil,nil}}},0}}]},{trap_exit,true},{status,running},{heap_size,2586},{stack_size,28},{reductions,23098}], [{neighbour,[{pid,<0.1542.0>},{registered_name,[]},{initial_call,{gm,init,['Argument__1']}},{current_function,{gen_server2,process_next_msg,1}},{ancestors,[<0.1541.0>,<0.1539.0>,<0.1538.0>,<0.710.0>,<0.694.0>,<0.693.0>,rabbit_vhost_sup_sup,rabbit_sup,<0.272.0>]},{message_queue_len,0},{links,[<0.1541.0>]},{trap_exit,false},{status,waiting},{heap_size,376},{stack_size,10},{reductions,33181},{current_stacktrace,[{gen_server2,process_next_msg,1,[{file,"src/gen_server2.erl"},{line,673}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}]},{neighbour,[{pid,<0.1541.0>},{registered_name,[]},{initial_call,{rabbit_mirror_queue_coordinator,init,['Argument__1']}},{current_function,{erlang,hibernate,3}},{ancestors,[<0.1539.0>,<0.1538.0>,<0.710.0>,<0.694.0>,<0.693.0>,rabbit_vhost_sup_sup,rabbit_sup,<0.272.0>]},{message_queue_len,0},{links,[<0.1539.0>,<0.1542.0>]},{trap_exit,false},{status,waiting},{heap_size,258},{stack_size,1},{reductions,3627},{current_stacktrace,[]}]}]

2021-03-21 18:40:10.875 [error] <0.1539.0> CRASH REPORT Process <0.1539.0> with 2 neighbours exited with reason: no match of right hand value [{<0.1535.0>,<<"amq.ctag-inTYKurUD8OQ1NAJyjK_Jw">>,true,250,true,up,[],<<"infra">>}] in rabbit_amqqueue_process:handle_cast/2 line 1637 in gen_server2:terminate/3 line 1183

2021-03-21 18:40:10.875 [info] <0.1538.0> supervisor: {<0.1538.0>,rabbit_amqqueue_sup}, errorContext: child_terminated, reason: {{badmatch,[{<0.1535.0>,<<"amq.ctag-inTYKurUD8OQ1NAJyjK_Jw">>,true,250,true,up,[],<<"infra">>}]},[{rabbit_amqqueue_process,handle_cast,2,[{file,"src/rabbit_amqqueue_process.erl"},{line,1637}]},{gen_server2,handle_msg,2,[{file,"src/gen_server2.erl"},{line,1067}]},{proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,236}]}]}, offender: [{pid,<0.1539.0>},{id,rabbit_amqqueue},{mfargs,{rabbit_prequeue,start_link,[{amqqueue,{resource,<<"/dev">>,queue,<<"Test-My-Lovely-SAC">>},true,false,none,[{<<"x-single-active-consumer">>,bool,true}],none,[],[],[],[{vhost,<<"/dev">>},{name,<<"ha-mode">>},{pattern,<<".*">>},{'apply-to',<<"all">>},{definition,[{<<"ha-mode">>,<<"exactly">>},{<<"ha-params">>,3},{<<"ha-sync-mode">>,<<"automatic">>}]},{priority,0}],undefined,[],[],live,0,[],<<"/dev">>,#{user => <<"infra">>},rabbit_classic_queue,#{}},declare,<0.1537.0>]}},{restart_type,intrinsic},{shutdown,30000},{child_type,worker}]

2021-03-21 18:40:10.876 [error] <0.1538.0> Supervisor {<0.1538.0>,rabbit_amqqueue_sup} had child rabbit_amqqueue started with rabbit_prequeue:start_link({amqqueue,{resource,<<"/dev">>,queue,<<"Test-My-Lovely-SAC">>},true,false,none,[{<<"x-single-act...">>,...}],...}, declare, <0.1537.0>) at <0.1539.0> exit with reason no match of right hand value [{<0.1535.0>,<<"amq.ctag-inTYKurUD8OQ1NAJyjK_Jw">>,true,250,true,up,[],<<"infra">>}] in rabbit_amqqueue_process:handle_cast/2 line 1637 in context child_terminated

2021-03-21 18:40:10.892 [info] <0.636.0> rabbit on node rabbit@rabbitmq2 up