maxSegmentsToMove

[Chris Lee]

Is there a formula or some guidelines regarding how to set maxSegmentsToMove in the coordinator's dynamic config? I'm thinking about increasing it so when new historical nodes are brought up, the segments are loaded quicker. However, I'm also worried that setting the the value too high will cause constant rebalancing that will bog down the cluster.

I currently just have it set to the default value of 5. Saw some posts suggesting increasing it to 500, but couldn't find any specific guidelines.

[Ben Krug]

There are maxSegmentsToMove and maxSegmentsInNodeLoadingQueue (see this page).

To speed up loading when new nodes are brought up, you also want to look at the latter.  (And there are very general recommendations given on that page.)

I don't have a general guideline, but sometimes people bump them up high when bringing up nodes, then reduce again.

You can monitor segment movement and loading when adding nodes, or normally, in the druid console.  You don't want it 

unnecessarily moving segments back and forth all the time, but you also don't want it idle when bringing up new nodes.

On Fri, Apr 8, 2022 at 3:17 PM Chris Lee <chri...@autonomic.ai> wrote:

Is there a formula or some guidelines regarding how to set maxSegmentsToMove in the coordinator's dynamic config? I'm thinking about increasing it so when new historical nodes are brought up, the segments are loaded quicker. However, I'm also worried that setting the the value too high will cause constant rebalancing that will bog down the cluster.

I currently just have it set to the default value of 5. Saw some posts suggesting increasing it to 500, but couldn't find any specific guidelines.

--
You received this message because you are subscribed to the Google Groups "Druid User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/78807d67-7ce3-4f25-8baa-d6862c15d7edn%40googlegroups.com.