Clients cannot connect when using Proxy Protocol with F5 LTM

[David Diehl]

I am using RabbitMQ 3.7.6 and Erlang 20.2.3.

I've configured the F5 LTM per this direction:  https://groups.google.com/forum/#!topic/rabbitmq-users/Sc42wWHuXFw

My rabbitmq configuration file is /etc/rabbitmq/rabbitmq.conf and contains the following:

## Whether or not to enable proxy protocol support.

## Once enabled, clients cannot directly connect to the broker

## anymore. They must connect through a load balancer that sends the

## proxy protocol header to the broker at connection time.

## This setting applies only to AMQP clients, other protocols

## like MQTT or STOMP have their own setting to enable proxy protocol.

## See the plugins documentation for more information.

##

proxy_protocol = true

I verified that it is using this configuration file, as the log file contains this snippet during startup:

 home dir       : /var/lib/rabbitmq

 config file(s) : /etc/rabbitmq/rabbitmq.conf

 cookie hash    : GOeNNW1a8SksMTlXvAt2oQ==

I have approximately 90 connections to the rabbit server.

Of these client connections about 40 are shovels on other 3.7.6 (and Erlang 20.2.3 also) RabbitMQ servers and about 15 clients are Java Spring AMQP via Spring Boot version 1.5.10.RELEASE.  All of these 55 connections are using the F5 VIP and are therefore proxied.  Admittedly, I have an additional 35 shovel connections from a mixture of RabbitMQ version 3.6.10 and version 3.4.4 servers that are not (yet) configured to route through the load balancer; that is to say, they fail to connect because (and I realize this) that the proxy protocol says once activated all clients must go through the proxy.  I just mention this in case you think it could be part of my problem (although I wouldn't know why).

So, the behavior I see is very strange.  If the Rabbit server is running in this configuration, and new clients try to connect via the F5 proxy, I see entries like these in the rabbit log file:

2018-07-05 09:25:26.197 [info] <0.32035.0> accepting AMQP connection <0.32035.0> (10.5.130.35:50616 -> 10.5.130.15:5672)

2018-07-05 09:25:26.208 [warning] <0.32035.0> closing AMQP connection <0.32035.0> (10.5.130.35:50616 -> 10.5.130.15:5672):

client unexpectedly closed TCP connection

Meanwhile, the Spring Boot clients log these lines on every connection attempt (I have it configured to re-try every 15 seconds):

2018-07-05 09:25:25,220|INFO||SimpleAsyncTaskExecutor-1|org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer|||||Restarting Consumer@31a85f60: tags=[{}], channel=null, acknowledgeMode=AUTO local queue size=0

2018-07-05 09:25:25,227|INFO||SimpleAsyncTaskExecutor-2|org.springframework.amqp.rabbit.connection.CachingConnectionFactory|||||Attempting to connect to: tstsrvr.mycompanynamehere:5672

2018-07-05 09:25:25,227|WARN||AMQP Connection 10.5.120.47:5672|com.rabbitmq.client.impl.ForgivingExceptionHandler|||||An unexpected connection driver error occured (Exception message: Socket closed)

However, while the clients are trying to connect like this, repeatedly trying every so many seconds, if I restart the RabbitMQ server, all of the clients that were previously unable to connect, are able to connect!  It does seem to take them a bit longer to connect than normal (with the proxy protocol turned on), but they do connect successfully just as the rabbit server starts up.

Have you ever heard of something like this?  I have another server in this exact same configuration (same RabbitMQ and Erlang version too) and I don't see this behavior with that server.  This other server has fewer total clients (43) and the other difference is that I don't have "bad" clients trying to connect to it directly (rather than going through the proxy).

I guess another question I have is what should I expect to see if a client tries to connect directly to a server that is configured with "proxy_protocol = true"?  Should I see any connection attempt being logged?

I was just experimenting and set "proxy_protocol = false" but left the F5 configured with the proxyheader iRule and the Rabbit server very nicely logged these types of messages:

2018-07-05 15:51:34.112 [info] <0.29777.0> accepting AMQP connection <0.29777.0> (10.5.130.250:53782 -> 10.5.130.15:5672)

2018-07-05 15:51:34.118 [warning] <0.29777.0> closing AMQP connection <0.29777.0> (10.5.130.250:53782 -> 10.5.130.15:5672):

{bad_header,<<"PROXY TC">>}

So I clearly see what doing something like that will produce.  But it leaves me wondering what I should see if a direct connection is attempted when proxy protocol is true.

thanks..  dave

[Michael Klishin]

Since no RabbitMQ client or Shovels support Proxy Protocol, your load balancer must do that. Which may or may not be

enabled by default, for example.

You have two types of warnings posted here. One is covered in [1]. Another means that the target RabbitMQ version

DOES NOT support Proxy Protocol and does not recognize the PROXY TCP … header F5 seems to be sending.

Proxy Protocol is supported starting with RabbitMQ 3.7.0 [2].

1. http://www.rabbitmq.com/logging.html#logged-events

2. http://www.rabbitmq.com/changelog.html

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ