RabbitMQ streams are availalbe for consumers but not producers

[Nicolas Piguet]

Hello,

I once again have a weird situation for my RMQ streams following a network split.

The admin interface shows the following status:

It seems that consumers are able to connect to all streams (and the admin UI shows a bunch of consumers connected to each stream), however producers cannot connect to the streams named s-sr0-119 and s-sr240-339. We are using the RMQ streams java client library and it returns the following error:

  java.lang.IllegalStateException: Not leader available for stream oems.trigger.events.s-sr0-119.q

Since we've previous had similar (but not identical) issues with mnesia update tasks becoming stuck after a network partition, I've tried running the command described here again https://groups.google.com/g/rabbitmq-users/c/Ta3zC-RcF7U/m/jj-bVAYEAQAJ  , but it did not resolve the issue.

The cluster has 3 nodes (04,05,06). The server-side logs on node 05 look pretty normal, but the logs on nodes 04 and 06 show a lot of suspicious lines that say:

    [warning] <0.5248.13> Error when retrieving broker metadata: <<"pre-bpubsub05glc">> undefined

And sometimes blocks that read:

   [warning] <0.13757.1> Management delegate query returned errors:
   [warning] <0.13757.1> [{<13046.9439.1>,
   [warning] <0.13757.1>   {error,badarg,
   [warning] <0.13757.1>          [{ets,select,
   [warning] <0.13757.1>                [rabbit_stream_publisher_created,
   [warning] <0.13757.1>                 [{{{{resource,'$1','_','$2'},'_','_'},'_'},
   [warning] <0.13757.1>                   [{'andalso',{'==','$1',<<"scs.securities-brokerage">>},
   [warning] <0.13757.1>                               {'==',<<"oems.trigger.events.s-sr120-239.q">>,
   [warning] <0.13757.1>                                     '$2'}}],
   [warning] <0.13757.1>                   ['$_']}]],
   [warning] <0.13757.1>                [{error_info,#{cause => id,module => erl_stdlib_errors}}]},
   [warning] <0.13757.1>           {rabbit_stream_mgmt_db,get_entity_stats_by_resource,2,
   [warning] <0.13757.1>                                  [{file,"rabbit_stream_mgmt_db.erl"},
   [warning] <0.13757.1>                                   {line,142}]},
   [warning] <0.13757.1>           {rabbit_stream_mgmt_db,entity_data,4,
   [warning] <0.13757.1>                                  [{file,"rabbit_stream_mgmt_db.erl"},
   [warning] <0.13757.1>                                   {line,92}]},
   [warning] <0.13757.1>           {delegate,safe_invoke,2,[{file,"delegate.erl"},{line,267}]},
   [warning] <0.13757.1>           {delegate,'-safe_invoke/2-lc$^0/1-0-',2,
   [warning] <0.13757.1>                     [{file,"delegate.erl"},{line,264}]},
   [warning] <0.13757.1>           {delegate,handle_call,3,[{file,"delegate.erl"},{line,282}]},
   [warning] <0.13757.1>           {gen_server2,handle_msg,2,[{file,"gen_server2.erl"},{line,1046}]},
   [warning] <0.13757.1>           {proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,250}]}]}}]

I unfortunately do not have the ability to turn on debug logs.

Do anyone have an idea of what is going on, and how I can get my streams running again?

Thanks for your help

Software versions:
    - RabbitMQ: 3.10.8
    - Erlang: 25.0.4
    - RabbitMQ streams client java: 0.9.0

[Nicolas Piguet]

Ok, further checks reveal that the big error block that starts with " Management delegate query returned errors:" occurs every 5 seconds when the admin UI is open on any of the Stream details page. Still weird, but not too critical, I guess.

The other line ("Error when retrieving broker metadata: <<"pre-bpubsub05glc">> undefined") occurs twice in a row, every 5 seconds but doesn't seem to be linked to the admin UI.

[Arnaud Cogoluègnes]

The cause of the "Error when retrieving broker metadata: <<"pre-bpubsub05glc">> undefined" is the failure to get stream information from all nodes [1]. In this case the stream plugin port of the pre-bpubsub05glc node is undefined. Is the stream plugin properly started on this node?

[1] https://github.com/rabbitmq/rabbitmq-server/blob/3432dcb1612dfa403a970b23cc7e099b6b8e3293/deps/rabbitmq_stream/src/rabbit_stream_reader.erl#L2397

[Nicolas Piguet]

Just adding a response here for future reference: Arnaud was right. For some reason, the problematic node of our cluster had a different set of enabled plugins, and both rabbitmq_stream and rabbitmq_stream_management were missing.

[Arnaud Cogoluègnes]

OK, thanks for reporting back!