MQTT and AWS ELB : How to make ELB redirect connection requests from existing clients to active node

[Shobhana Sriram]

We are trying to switch from ActiveMQ to RabbitMQ and I am performing some tests in this regard using the MQTT plugin with all default configurations.

I have a cluster of 2 RabbitMQ nodes (each running version 3.6.10 of RabbitMQ with MQTT plugin enabled) and an AWS classic load balancer in front of them.

Clients (apps running on mobile devices and using Eclipse Paho client lib) connect to the cluster and the load balancer distributes connections in round-robin fashion.

When I bring down one node, say Node1, all clients that were connected to Node1 get a callback indicating connection to the broker is lost.

These clients try to reconnect but the connection attempt fails indicating the broker is not reachable.

From AWS console I can see that AWS ELB detects that Node1 is down and marks it as "OutOfService".

Connection requests from new clients are routed to the "InService" node Node2; however, connection requests from existing clients that were previously connected to Node1 always fail!

ELB is configured with idle timeout of 180 seconds. Enabling or disabling connection draining in ELB did not make any difference.

Is there any specific configuration to make ELB *forget* that the existing clients were connected to Node1 and allow them to connect to Node2?

[Michael Klishin]

AWS ELB, HAproxy and pretty much every load balancer out there tracks whether backends

are available using health checks or timeouts.

As soon as load balancer detects that a backend is down, new connections won't be routed to it

but it does not happen in an instant.

I can't find a specific AWS ELB doc guide about health checks but this page has a table that

suggests they are supported:

http://docs.aws.amazon.com/elasticloadbalancing/latest/userguide/what-is-load-balancing.html

This guide has some decent overall advice on healthcheck use:

https://aws.amazon.com/articles/1636185810492479

This should not be confused with *idle* connection timeouts:

https://aws.amazon.com/blogs/aws/elb-idle-timeout-control/

http://docs.aws.amazon.com/elasticloadbalancing/latest/classic/config-idle-timeout.html

which can lead to the opposite issue: connections to available backends being closed for

no good reason. Heartbeats (keep-alives in MQTT) is one way of working it around on mostly idle connections:

http://rabbitmq.com/heartbeats.html

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Shobhana Sriram]

Thank you MK, for your quick response and additional links.

I have gone through these links and have a fair understanding of the tips/advice given in these links, but could not understand what is wrong with my setup since the session stickiness is not available for TCP protocol! Also, new connections with new client IDs (from same mobile devices though) were getting routed to the active node; only connection attempts from existing client IDs were failing since they were still getting routed to inactive node.

Then I tried by adding following HA policy :

rabbitmqctl set_policy ha-mqtt "^mqtt" \ '{"ha-mode":"exactly","ha-params":2,"ha-sync-mode":"automatic"}'

With this policy in place, all queues created for MQTT clients were mirrored. Now when Node1 is down, connection attempts from existing client IDs also get routed to the other active node!

This makes me wonder what is the relationship between client IDs from MQTT clients and their connection to broker nodes? I thought mirroring of queues is necessary only to retain and access messages that were not yet acknowledged when the queue master node goes down. But I see that the clients are not even able to establish a connection!

Any links to any documents that help me understand this relationship is appreciated.

TIA,

Shobhana

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.