Retry request for shard [96] homes from coordinator at [Actor[akka.tcp://[]/system/sharding/myActorCoordinator/singleton/coordinator#-139056758]]. [xYZ] buffered messages.

[Ganga Reddy]

I'm injecting actors into my 50 node cluster through a service actor which reads from a stream and creates actors based on data in streams.  

It seems that all the shards of the actors are being forced on to the same node. The logs are here, 

19:24:53.832 [Segmenter-akka.actor.default-dispatcher-38] DEBUG s.b.segmenter.streams.CustomerLoader - Started a source with seed value

[INFO] [03/29/2018 19:24:56.650] [Segmenter-akka.actor.default-dispatcher-37] [akka.tcp://Segm...@10.244.125.8:2551/system/sharding/customerActor] ShardRegion for [customerActor] is using [10.0 %] of its buffer capacity.

[INFO] [03/29/2018 19:25:00.038] [Segmenter-akka.actor.default-dispatcher-2] [akka.tcp://Segm...@10.244.125.8:2551/system/sharding/customerActor] ShardRegion for [customerActor] is using [20.0 %] of its buffer capacity.

[INFO] [03/29/2018 19:25:01.908] [Segmenter-akka.actor.default-dispatcher-36] [akka.tcp://Segm...@10.244.125.8:2551/system/sharding/customerActor] ShardRegion for [customerActor] is using [30.0 %] of its buffer capacity.

[INFO] [03/29/2018 19:25:03.283] [Segmenter-akka.actor.default-dispatcher-5] [akka.tcp://Segm...@10.244.125.8:2551/system/sharding/customerActor] ShardRegion for [customerActor] is using [40.0 %] of its buffer capacity.

[INFO] [03/29/2018 19:25:07.808] [Segmenter-akka.actor.default-dispatcher-2] [akka.tcp://Segm...@10.244.125.8:2551/system/sharding/customerActor] ShardRegion for [customerActor] is using [50.0 %] of its buffer capacity.

[WARN] [03/29/2018 19:25:08.815] [Segmenter-akka.actor.default-dispatcher-3] [akka.tcp://Segm...@10.244.125.8:2551/system/sharding/customerActor] ShardRegion for [customerActor] is using [60.0 %] of its buffer capacity. The coordinator might not be available. You might want to check cluster membership status.

[WARN] [03/29/2018 19:25:10.306] [Segmenter-akka.actor.default-dispatcher-3] [akka.tcp://Segm...@10.244.125.8:2551/system/sharding/customerActor] ShardRegion for [customerActor] is using [70.0 %] of its buffer capacity. The coordinator might not be available. You might want to check cluster membership status.

[WARN] [03/29/2018 19:25:11.265] [Segmenter-akka.actor.default-dispatcher-6] [akka.tcp://Segm...@10.244.125.8:2551/system/sharding/customerActor] ShardRegion for [customerActor] is using [80.0 %] of its buffer capacity. The coordinator might not be available. You might want to check cluster membership status.

[WARN] [03/29/2018 19:25:14.693] [Segmenter-akka.actor.default-dispatcher-2] [akka.tcp://Segm...@10.244.125.8:2551/system/sharding/customerActor] ShardRegion for [customerActor] is using [80.0 %] of its buffer capacity. The coordinator might not be available. You might want to check cluster membership status.

[WARN] [03/29/2018 19:25:15.781] [Segmenter-akka.actor.default-dispatcher-6] [akka.tcp://Segm...@10.244.125.8:2551/system/sharding/customerActor] ShardRegion for [customerActor] is using [90.0 %] of its buffer capacity. The coordinator might not be available. You might want to check cluster membership status.

[WARN] [03/29/2018 19:25:17.348] [Segmenter-akka.actor.default-dispatcher-37] [akka.tcp://Segm...@10.244.125.8:2551/system/sharding/customerActor] ShardRegion for [customerActor] is using [100.0 %] of its buffer capacity. The coordinator might not be available. You might want to check cluster membership status.

[WARN] [03/29/2018 19:25:17.348] [Segmenter-akka.actor.default-dispatcher-37] [akka.tcp://Segm...@10.244.125.8:2551/system/sharding/customerActor] Buffer is full, dropping message for shard [161]

[INFO] [03/29/2018 19:25:17.349] [Segmenter-akka.actor.default-dispatcher-6] [akka://Segmenter/deadLetters] Message [starbucks.bits.segmenter.models.Customer] from Actor[akka://Segmenter/system/sharding/customerActor#-283448532] to Actor[akka://Segmenter/deadLetters] was not delivered. [2] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.

[INFO] [03/29/2018 19:25:17.356] [Segmenter-akka.actor.default-dispatcher-2] [akka://Segmenter/deadLetters] Message [starbucks.bits.segmenter.models.Customer] from Actor[akka://Segmenter/system/sharding/customerActor#-283448532] to Actor[akka://Segmenter/deadLetters] was not delivered. [3] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.

[INFO] [03/29/2018 19:25:17.365] [Segmenter-akka.actor.default-dispatcher-38] [akka://Segmenter/deadLetters] Message [starbucks.bits.segmenter.models.Customer] from Actor[akka://Segmenter/system/sharding/customerActor#-283448532] to Actor[akka://Segmenter/deadLetters] was not delivered. [4] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.

[INFO] [03/29/2018 19:25:17.366] [Segmenter-akka.actor.default-dispatcher-36] [akka://Segmenter/deadLetters] Message [starbucks.bits.segmenter.models.Customer] from Actor[akka://Segmenter/system/sharding/customerActor#-283448532] to Actor[akka://Segmenter/deadLetters] was not delivered. [5] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.

[INFO] [03/29/2018 19:25:17.366] [Segmenter-akka.actor.default-dispatcher-36] [akka://Segmenter/deadLetters] Message [starbucks.bits.segmenter.models.Customer] from Actor[akka://Segmenter/system/sharding/customerActor#-283448532] to Actor[akka://Segmenter/deadLetters] was not delivered. [6] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.

[INFO] [03/29/2018 19:25:17.372] [Segmenter-akka.actor.default-dispatcher-36] [akka://Segmenter/deadLetters] Message [starbucks.bits.segmenter.models.Customer] from Actor[akka://Segmenter/system/sharding/customerActor#-283448532] to Actor[akka://Segmenter/deadLetters] was not delivered. [7] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.

[INFO] [03/29/2018 19:25:17.372] [Segmenter-akka.actor.default-dispatcher-36] [akka://Segmenter/deadLetters] Message [starbucks.bits.segmenter.models.Customer] from Actor[akka://Segmenter/system/sharding/customerActor#-283448532] to Actor[akka://Segmenter/deadLetters] was not delivered. [8] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.

[INFO] [03/29/2018 19:25:17.372] [Segmenter-akka.actor.default-dispatcher-36] [akka://Segmenter/deadLetters] Message [starbucks.bits.segmenter.models.Customer] from Actor[akka://Segmenter/system/sharding/customerActor#-283448532] to Actor[akka://Segmenter/deadLetters] was not delivered. [9] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.

[INFO] [03/29/2018 19:25:17.372] [Segmenter-akka.actor.default-dispatcher-5] [akka://Segmenter/deadLetters] Message [starbucks.bits.segmenter.models.Customer] from Actor[akka://Segmenter/system/sharding/customerActor#-283448532] to Actor[akka://Segmenter/deadLetters] was not delivered. [10] dead letters encountered, no more dead letters will be logged. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.

[WARN] [03/29/2018 19:25:18.422] [Segmenter-akka.actor.default-dispatcher-36] [akka.tcp://Segm...@10.244.125.8:2551/system/sharding/customerActor] ShardRegion for [customerActor] is using [100.0 %] of its buffer capacity. The coordinator might not be available. You might want to check cluster membership status.

[WARN] [03/29/2018 19:25:18.422] [Segmenter-akka.actor.default-dispatcher-36] [akka.tcp://Segm...@10.244.125.8:2551/system/sharding/customerActor] Buffer is full, dropping message for shard [325]

[WARN] [03/29/2018 19:25:19.269] [Segmenter-akka.actor.default-dispatcher-38] [akka.tcp://Segm...@10.244.125.8:2551/system/sharding/customerActor] ShardRegion for [customerActor] is using [100.0 %] of its buffer capacity. The coordinator might not be available. You might want to check cluster membership status.

[WARN] [03/29/2018 19:25:19.274] [Segmenter-akka.actor.default-dispatcher-38] [akka.tcp://Segm...@10.244.125.8:2551/system/sharding/customerActor] Buffer is full, dropping message for shard [163]

[WARN] [03/29/2018 19:25:20.463] [Segmenter-akka.actor.default-dispatcher-6] [akka.tcp://Segm...@10.244.125.8:2551/system/sharding/customerActor] ShardRegion for [customerActor] is using [100.0 %] of its buffer capacity. The coordinator might not be available. You might want to check cluster membership status.

[WARN] [03/29/2018 19:25:20.463] [Segmenter-akka.actor.default-dispatcher-6] [akka.tcp://Segm...@10.244.125.8:2551/system/sharding/customerActor] Buffer is full, dropping message for shard [319]

[WARN] [03/29/2018 19:25:21.242] [Segmenter-akka.actor.default-dispatcher-2] [akka.tcp://Segm...@10.244.125.8:2551/system/sharding/customerActor] Buffer is full, dropping message for shard [99]

[WARN] [03/29/2018 19:25:21.355] [Segmenter-akka.actor.default-dispatcher-5] [akka.tcp://Segm...@10.244.125.8:2551/system/sharding/customerActor] ShardRegion for [customerActor] is using [100.0 %] of its buffer capacity. The coordinator might not be available. You might want to check cluster membership status.