Long migration times when nodes are added or removed

[Ben]

Hello,

We are running a Hazelcast 3.8.7 cluster of 55 nodes with 28G heap each within a single AWS region across 3 AZs (no placement groups).  Appropriate instance types have been chosen based on the recommendations on https://github.com/hazelcast/hazelcast-aws.  We have configured a partition count of 4001 to accommodate for future scaling of the cluster, resulting in 72 partitions per node.  We have a single IMap, that ranges between ~0.5 million up to ~1.8 million entries.  Used heap can climb to 22GB.  We see issues with long migration times when adding or removing nodes from the cluster, especially during peak traffic times.  Sometimes it takes over an hour for the cluster to return to safe state when a node is added/removed.  Other times it may take only 5 minutes.  When long migration occurs, we see upwards of 15k migration tasks that appear to execute very slowly.  We've been combing the documentation tweaks or suggestions on how to speed migration time but haven't seen anything definitive.  Is this expected behavior and are there any tweaks or settings that can speed up the migration process?

Thanks

[Sertuğ Kaya]

Hello Ben;

You have quite a cluster there :) 

Whenever you add/remove a member, partitions are migrated between members to rebalance the data distribution. During that it is important to make sure you don't have partitions very large. Considering you have roughly (55*22 GB) ~1.2TB of data, your partition size is currently (1.2TB/4001) ~300MB, which is considered larger than the best practice, it's suggested to keep partition sizes under 100MB, or around 50MB if possible. In order to do that, you can triple your current partition count at least. This would help on reducing the time consumed when sending a partition over the network.

Other than that, I'm guessing you might have some GC issues with a 22GB used heap, so I'd look at hazelcast logs or GC logs for any activity that might slow members down in general. 

On the other hand Hazelcast has off-heap storage option in the enterprise version which can eliminate the GC concern.

[David Brimley]

In addition to Sertugs excellent advice which you should definatley follow, you might also want to look at the latest version of Hazelcast 3.9.  This includes a new feature that breaks up the migraion of partitions into smaller chunks, increasing availability of your system.

"Gigantic Cache Migration Enhancements: It enables manual control on the partition migrations. Partitions can be migrated/replicated in small fragments. Please refer to the hazelcast.partition.migration.fragments.enabled system property"

http://docs.hazelcast.org/docs/rn/3.9.html

[Ben]

Thanks for the tips.  We will be increasing our partition count.  Based on our metrics, size per partition is hovering around 250MB on average.  We also realized that we were unnecessarily calling getLifecycleService().isRunning() before callling shutdown() - that seems to have made some improvement when scaling down.

[Ben]

Thanks, I will look into that.