Too many direct federation connections

[Selim Tuvi]

Hi this was supposed to be fixed a while back with this change:

https://github.com/rabbitmq/rabbitmq-federation/pull/77

But we are still seeing direct federation connections being left open. Currently one of our QA instances has ~ 20K federation connections open, some from about 10 days ago.

Here is the version we are at:

RabbitMQ 3.7.14

Erlang 21.3.7

We are running RabbitMQ inside a docker container on a VM. Docker version is 18.06.3-ce. VM is running CentOS 7.4.

Let us know if we can provide additional information.

Thanks

-Selim

[Selim Tuvi]

Not sure if it will help but we have now updated our upstream URIs and testing it with the following configuration:

amqp://{user}:{password}@{upstream-host}?heartbeat=10&connection_timeout=10000 

-Selim

[Selim Tuvi]

Also when an upstream node is restarted we are seeing the following in the log and the "Direct 0-9-1" connection count increases:

2019-06-10 18:50:49.724 [error] <0.9994.222> ** Generic server <0.9994.222> terminating

** Last message in was {'DOWN',#Ref<0.1034073456.3351248897.201279>,process,<0.10443.222>,shutdown}

** When Server state == {state,{amqqueue,{resource,<<"/">>,queue,<<"mercury.ilm-lon.source_check">>},true,false,none,[],<0.4180.0>,[],[],[],[{vhost,<<"/">>},{name,<<"federated-mercury-queues">>},{pattern,<<"^mercury\\..*">>},{'apply-to',<<"queues">>},{definition,[{<<"federation-upstream-set">>,<<"all">>}]},{priority,0}

],undefined,[],[rabbit_federation_queue],live,0,[],<<"/">>,#{user => <<"iris">>}},false,<0.10180.222>,<0.10443.222>,<0.10079.222>,<0.10062.222>,{upstream,[<<"amqp://iris:[redacted]@pe-mercuryservices-sing-02v">>],<<"mercury.ilm-lon.source_check">>,<<"mercury.ilm-lon.source_check">>,1000,1,5,none,none,false,'on-confirm',none,

<<"pe-mercuryservices-sing-02v">>,false},{upstream_params,<<"amqp://iris:[redacted]@pe-mercuryservices-sing-02v">>,{amqp_params_network,<<"iris">>,<<"[redacted]">>,<<"/">>,

"pe-mercuryservices-sing-02v",undefined,2047,0,10,60000,none,[#Fun<amqp_uri.12.79294410>,#Fun<amqp_uri.12.79294410>],[],[]},{amqqueue,{resource,<<"/">>,queue,<<

"mercury.ilm-lon.source_check">>},true,false,none,[],<0.4180.0>,[],[],[],[{vhost,<<"/">>},{name,<<"federated-mercury-queues">>},{pattern,<<"^mercury\\..*">>},{'apply-to',<<"queues">>},{definition,[{<<"federation-upstream-set">>,<<"all">>}]},{priority,0}],undefined,[],[rabbit_federation_queue],live,0,[],<<"/">>,#{user => <<"iris">>}},<<"amqp://pe-mercuryservices-sing-02v">>,[{<<"uri">>,longstr,<<"amqp://pe-mercuryservices-sing-02v">>},{<<"queue">>,longstr,<<"mercury.ilm-lon.source_check">>}]},{0,nil}}

** Reason for termination == 

** {upstream_channel_down,shutdown}

2019-06-10 18:50:49.734 [error] <0.9994.222> CRASH REPORT Process <0.9994.222> with 0 neighbours exited with reason: {upstream_channel_down,shutdown} in gen_server2:terminate/3 line 1172

2019-06-10 18:50:49.716 [error] <0.13418.222> Federation link could not create a disposable (one-off) channel due to an error exit: {noproc,{gen_server,call,[<0

.13415.222>,{command,{open_channel,none,{amqp_selective_consumer,[]}}},60000]}}

[Luke Bakken]

Hi Selim,

Could you please attach the entire upstream log file, or at least a file containing more log context?

Thanks!

Luke