Auto rebalancing of leaders

245 views
Skip to first unread message

Karl Stoney

unread,
Jun 16, 2023, 6:51:01 AM6/16/23
to rabbitmq-users
Hi Everyone, 
I'm relatively new to running rabbitmq; but it's going pretty well. 
Running on top of kubernetes (manual chart, not operator)

I'm been doing some testing and can see that following rolling restarts, typically leaders are left imbalanced across the cluster (the last node to restart for example won't have any leaders).

I know that I can trigger a manual leader rebalance (which i could do post-rollout) but i'm wondering if there's some better way to do this.

The other route i was thinking was writing an operator to go by the side of it that watches for pod events and runs a rebalance whenever a pod enters the ready state?

Any input is welcome!
Thanks

Michal Kuratczyk

unread,
Jun 16, 2023, 7:06:08 AM6/16/23
to rabbitm...@googlegroups.com
Hi,

Why don't you just use the Operator we developed? It already performs rebalancing after a restart. :)

Best,

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/da07127c-ff6f-4717-820b-1ac32f31c49cn%40googlegroups.com.


--
Michał
RabbitMQ team

Karl Stoney

unread,
Jun 16, 2023, 7:27:08 AM6/16/23
to rabbitm...@googlegroups.com

Haha that’s a very long/deep conversation/philosophical conversation 😊

 

At a high level, we believe in having Infrastructure as Code (complete manifests) that are promoted through environments (via helm).  This allows us to perform automated checks on the manifests (think quality, linting, rule enforcement etc) in the pipelines before it gets near the cluster.  We have admission webhooks doing similar things to which in theory would block an operator doing something that doesn’t align with our policies, but that is generally failing too late in the pipeline (we prefer to push left, our checks can be run locally on the developers machine too).

 

We are more easily able to identify what has changed across releases with the templating approach, as we’re able to do differentials of the templated output (regression testing manifest changes).

 

Our pipelines don’t go green until all the resources defined in the helm chart are fully deployed and available, operators will be doing _additional things_ outside the scope of those manifests that our pipelines will be unaware of, making “was this deployment successful yes or no” more difficult to do in a technology agnostic way.

 

The other advantage to the above is we don’t need to run privileged workloads on the cluster that can make runtime modifications.

 

I do want to stress that this is just our methodology, and operators certainly suit a lot of people and tick a lot of boxes, they’re just not our cup of tea! 😃

 

Rebalancing leaders kind of feels like it should be a continual thing that’s platform agnostic?  Eg Kubernetes, vms, whatever, it’d be really nice if there was the ability to say “constantly do your best rabbitmq to keep the leaders evenly distributed”?

 

Perhaps that’s plugin territory? I dunno.  I’m still too knew to the domain to make any solid suggestions!

 

From: rabbitm...@googlegroups.com <rabbitm...@googlegroups.com> on behalf of Michal Kuratczyk <mkura...@gmail.com>
Date: Friday, 16 June 2023 at 12:06
To: rabbitm...@googlegroups.com <rabbitm...@googlegroups.com>
Subject: Re: [rabbitmq-users] Auto rebalancing of leaders

You don't often get email from mkura...@gmail.com. Learn why this is important


Unless expressly stated otherwise in this email, this e-mail is sent on behalf of Auto Trader Limited Registered Office: 1 Tony Wilson Place, Manchester, Lancashire, M15 4FN (Registered in England No. 03909628). Auto Trader Limited is part of the Auto Trader Group Plc group. This email and any files transmitted with it are confidential and may be legally privileged, and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender. This email message has been swept for the presence of computer viruses.

Karl Stoney

unread,
Jun 16, 2023, 7:28:56 AM6/16/23
to rabbitm...@googlegroups.com

The amount of spelling and grammatical errors in my response is embarrassing 😃

Michal Kuratczyk

unread,
Jun 16, 2023, 7:45:30 AM6/16/23
to rabbitm...@googlegroups.com
Periodic rebalance is something that we discussed but at least for the time being, we decided against it.
Users tend to dislike things happening automatically if those things can affect their SLAs, performance, etc.
In fact, it sounds like you're very much in that group. :)

In some environments, queues are often deleted and declared - in those cases, they are "rebalanced", since
a node is picked for every declaration, according to the leader locator strategy.

That leaves us with automatic rebalancing after a restart. Here the problem is that RabbitMQ doesn't really know
what's being done to it. It could perform a rebalance after each node's restart / cluster (re)join, but then you'd
have redundant rebalances. It doesn't really know when all nodes that were supposed to be restarted, are now restarted.
It could try to guess but it could guess wrong and surprise you at the worst time.

At the orchestration layer however, it's clear - you updated the image for example, we know all nodes need to be restarted,
and then when they are all restarted, we can trigger a rebalance. To me, this sounds like the best place to do it.

The CLI and API endpoint (/api/rebalance/queues) are available to those who want to trigger a rebalance in other ways / at other times.

Best,



--
Michał
RabbitMQ team

Karl Nilsson

unread,
Jun 16, 2023, 7:51:07 AM6/16/23
to rabbitm...@googlegroups.com

About rebalancing, if this is related to quorum queues, it isn't necessarily the case that having the leaders balanced or continuously rebalanced is better. Whether it is better or not depends so much on the load of individual queues and the locality of publishing connections. Neither of which are considered when it comes to rebalancing.

The act of rebalancing could cause some degree of disruption, latency spikes as publishers may need to resend messages sent to a member that is now a follower etc so doing this continuously would make the predictability of the cluster less.. predictable etc etc.

For quorum queues it is more important that their members are balanced over the clusters. Say in a 7 node cluster of 3 member quorum queues you'd wan't the number of members to be roughly equal on each node.




--
Karl Nilsson

Karl Stoney

unread,
Jun 16, 2023, 7:55:09 AM6/16/23
to rabbitm...@googlegroups.com
Thanks for the detailed reply, totally makes sense.

We have the ability to define post deploy hooks as part of our platform so for now I'll just pop the API rebalance endpoint in there.  

Unfortunately that won't cover other scenarios where pods are moved around.  We have a pretty ephemeral environment, pods get moved due to node upgrades, rebalancing, affinity rules and evictions etc.  I'm trying to resist putting rabbit on its own dedicated nodes that are more "stable" as a bit of chaos helps me to ensure were building something that's as resilient as possible and have a certain level of automation to handle these types of scenarios as robustly as possible. 

Side note: I last touched rabbitmq about 7 years ago.  It really has moved on a lot since then.  So many stateful workloads feel like square pegs in round holes when running on kubernetes but rabbit absolutely doesn't.  It's been a pleasure to use so far so thanks for your efforts!

Karl 

Sent: Friday, June 16, 2023 12:44:54 PM
Reply all
Reply to author
Forward
0 new messages