Shard coordinator issue

[Dilip A]

In our cluster configuration, we have 2 "backend" nodes (with akka.cluster.roles = ["master"]) and a "frontend" node. (with akka.cluster.roles = ["client"])

We want the shardcoordinator to always be placed on a backend node, but this is not happening

If they are started in the order -> backend, frontend, backend 

When the first backend node is shutdown gracefully, the shard coordinator is not moving over to the other backend node. We see below error 

"[ClusterSystem-akka.actor.default-dispatcher-4] WARN  a.e.s.Slf4jLogger$$anonfun$receive$1$$anonfun$applyOrElse$2.apply$mcV$sp 76 - Trying to register to coordinator at [Some(ActorSelection[Anchor(akka://ClusterSystem/), Path(/system/sharding/masterCoordinator/singleton/coordinator)])], but no acknowledgement. Total [1] buffered messages."

After the frontend node is stopped and started, the coordinator is moved correctly and we see no errors.

If they are started in the order -> backend, backend, frontend. There is no issue/error and the coordinator moves to the remaining backend node when the first one is shutdown.

We believe that shard coordinator is moving to the oldest alive node irrespective of the roles. 

For a similar error on this thread - https://groups.google.com/forum/#!topic/akka-user/VDsHEM2Sxx0, one of the suggestion was to ensure ClusterSharding was started on all nodes.

In Backend nodes it is started using 

"ClusterSharding.get(system).start(....)"

In Frontend nodes it is started using 

"ClusterSharding.get(system).startProxy(...)"

We are using Akka 2.4.7 with Java implementation.

[Patrik Nordwall]

How do you define the role when you start the ClusterSharding?

BTW, you should use latest version, i.e. 2.4.16

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscribe@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--

Patrik Nordwall
Akka Tech Lead
Lightbend -  Reactive apps on the JVM
Twitter: @patriknw

[Dilip A]

Thanks Patrik. This is how we are adding the role while starting the clustersharding.

ClusterShardingSettings settings = ClusterShardingSettings.create(system).withCoordinatorSingletonSettings(ClusterSingletonManagerSettings.create(system).withRole("master"));
ClusterSharding.get(system).start(..., ..., settings,..);

Is there something else we need to add?

Cluster client approach:

We also tried by connecting frontend node as ClusterClient instead of joining the cluster. Then the shardcoordinator was moving to the other backend node without any issues.

But we noticed performance degradation while using the Cluster client  compared to frontend node joining the same cluster. 

Is it due to all calls routing through receptionist. Is the no of receptionists limited to 1 or can we increase it? 

Per the discussion in this thread, https://groups.google.com/forum/#!topic/akka-user/LYbRjk_y0I0 is it better to go with the approach of joining the same cluster?

What is a better way to improve performance using cluster client?

On Thursday, January 26, 2017 at 5:30:31 AM UTC-8, Patrik Nordwall wrote:

How do you define the role when you start the ClusterSharding?

BTW, you should use latest version, i.e. 2.4.16

On Tue, Jan 24, 2017 at 12:32 AM, Dilip A <dilipr...@gmail.com> wrote:

In our cluster configuration, we have 2 "backend" nodes (with akka.cluster.roles = ["master"]) and a "frontend" node. (with akka.cluster.roles = ["client"])

We want the shardcoordinator to always be placed on a backend node, but this is not happening

If they are started in the order -> backend, frontend, backend 

When the first backend node is shutdown gracefully, the shard coordinator is not moving over to the other backend node. We see below error 

"[ClusterSystem-akka.actor.default-dispatcher-4] WARN  a.e.s.Slf4jLogger$$anonfun$receive$1$$anonfun$applyOrElse$2.apply$mcV$sp 76 - Trying to register to coordinator at [Some(ActorSelection[Anchor(akka://ClusterSystem/), Path(/system/sharding/masterCoordinator/singleton/coordinator)])], but no acknowledgement. Total [1] buffered messages."

After the frontend node is stopped and started, the coordinator is moved correctly and we see no errors.

If they are started in the order -> backend, backend, frontend. There is no issue/error and the coordinator moves to the remaining backend node when the first one is shutdown.

We believe that shard coordinator is moving to the oldest alive node irrespective of the roles. 

For a similar error on this thread - https://groups.google.com/forum/#!topic/akka-user/VDsHEM2Sxx0, one of the suggestion was to ensure ClusterSharding was started on all nodes.

In Backend nodes it is started using 

"ClusterSharding.get(system).start(....)"

In Frontend nodes it is started using 

"ClusterSharding.get(system).startProxy(...)"

We are using Akka 2.4.7 with Java implementation.

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.

To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.

To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

[Dilip A]

Hi Patrick,

Our cluster configuration has 2 "backend" nodes (akka.cluster.roles = ["master"]) and a "frontend" node.(akka.cluster.roles = ["client"])

The shardcoordinator is configured to be on a backend node, but this is not happening.

When the processes are started in the order backend, frontend and backend, and after a while if first backend node stopped, we notice that the frontend node is not working as expected.

We see from the logs of the 2nd backend node that the shard coordinators are moved to this node but the frontend node shows error that "trying to register coordinator at [Some(ActorSelection[Anchor(akka://ClusterSystem/), Path(/system/sharding/masterCoordinator/singleton/coordinator)])], but no acknowledgement. Total [1] buffered messages"

Are we missing some configuration that enables frontend node to know that the shard coordinators have moved to 2nd backend node?

Frontend node logs:

2017-02-02T18:33:14,996Z [ClusterSystem-akka.actor.default-dispatcher-29] ERROR a.e.s.Slf4jLogger$$anonfun$receive$1$$anonfun$applyOrElse$1.apply$mcV$sp 70 - AssociationError [akka.tcp://Cluste...@127.0.0.1:7123] -> [akka.tcp://Cluste...@127.0.0.1:8123]: Error [Shut down address: akka.tcp://Cluste...@127.0.0.1:8123] [
akka.remote.ShutDownAssociation: Shut down address: akka.tcp://Cluste...@127.0.0.1:8123
Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down.
]
2017-02-02T18:33:18,855Z [ClusterSystem-akka.actor.default-dispatcher-34] WARN  a.e.s.Slf4jLogger$$anonfun$receive$1$$anonfun$applyOrElse$2.apply$mcV$sp 78 - Association with remote system [akka.tcp://Cluste...@127.0.0.1:8123] has failed, address is now gated for [10] ms. Reason: [Association failed with [akka.tcp://Cluste...@127.0.0.1:8123]] Caused by: [The remote system explicitly disassociated (reason unknown).]
2017-02-02T18:33:19,571Z [ClusterSystem-akka.actor.default-dispatcher-34] INFO  a.e.s.Slf4jLogger$$anonfun$receive$1$$anonfun$applyOrElse$3.apply$mcV$sp 83 - Cluster Node [akka.tcp://Cluste...@127.0.0.1:7123] - Marking exiting node(s) as UNREACHABLE [Member(address = akka.tcp://Cluste...@127.0.0.1:8123, status = Exiting)]. This is expected and they will be removed.
2017-02-02T18:33:19,853Z [ClusterSystem-akka.actor.default-dispatcher-28] WARN  a.e.s.Slf4jLogger$$anonfun$receive$1$$anonfun$applyOrElse$2.apply$mcV$sp 78 - Association with remote system [akka.tcp://Cluste...@127.0.0.1:8123] has failed, address is now gated for [10] ms. Reason: [Association failed with [akka.tcp://Cluste...@127.0.0.1:8123]] Caused by: [The remote system explicitly disassociated (reason unknown).]




2017-02-02T18:33:52,759Z [ClusterSystem-akka.actor.default-dispatcher-27] WARN  a.e.s.Slf4jLogger$$anonfun$receive$1$$anonfun$applyOrElse$2.apply$mcV$sp 78 - Trying to register to coordinator at [Some(ActorSelection[Anchor(akka://ClusterSystem/), Path(/system/sharding/masterCoordinator/singleton/coordinator)])], but no acknowledgement. Total [1] buffered messages.
2017-02-02T18:33:53,759Z [ClusterSystem-akka.actor.default-dispatcher-27] WARN  a.e.s.Slf4jLogger$$anonfun$receive$1$$anonfun$applyOrElse$2.apply$mcV$sp 78 - Trying to register to coordinator at [Some(ActorSelection[Anchor(akka://ClusterSystem/), Path(/system/sharding/masterCoordinator/singleton/coordinator)])], but no acknowledgement. Total [1] buffered messages.

2nd backend node logs:

[ERROR] [02/02/2017 10:33:15.006] [ClusterSystem-akka.remote.default-remote-dispatcher-13] [akka.tcp://Cluste...@127.0.0.1:6123/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FClusterSystem%40127.0.0.1%3A8123-0/endpointWriter] AssociationError [akka.tcp://Cluste...@127.0.0.1:6123] -> [akka.tcp://Cluste...@127.0.0.1:8123]: Error [Shut down address: akka.tcp://Cluste...@127.0.0.1:8123] [
akka.remote.ShutDownAssociation: Shut down address: akka.tcp://Cluste...@127.0.0.1:8123
Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down.
]
[INFO] [02/02/2017 10:33:16.837] [ClusterSystem-akka.actor.default-dispatcher-28] [akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://Cluste...@127.0.0.1:6123] - Leader is moving node [akka.tcp://Cluste...@127.0.0.1:8123] to [Exiting]
[INFO] [02/02/2017 10:33:16.841] [ClusterSystem-akka.actor.default-dispatcher-21] [akka.tcp://Cluste...@127.0.0.1:6123/system/sharding/issueCoordinator] Younger observed OldestChanged: [Some(akka.tcp://Cluste...@127.0.0.1:8123) -> myself]
[INFO] [02/02/2017 10:33:16.841] [ClusterSystem-akka.actor.default-dispatcher-23] [akka.tcp://Cluste...@127.0.0.1:6123/system/sharding/shipment-issue-relationCoordinator] Younger observed OldestChanged: [Some(akka.tcp://Cluste...@127.0.0.1:8123) -> myself]
[INFO] [02/02/2017 10:33:16.841] [ClusterSystem-akka.actor.default-dispatcher-36] [akka.tcp://Cluste...@127.0.0.1:6123/system/sharding/locationCoordinator] Younger observed OldestChanged: [Some(akka.tcp://Cluste...@127.0.0.1:8123) -> myself]
[INFO] [02/02/2017 10:33:16.841] [ClusterSystem-akka.actor.default-dispatcher-28] [akka.tcp://Cluste...@127.0.0.1:6123/system/sharding/eventCoordinator] Younger observed OldestChanged: [Some(akka.tcp://Cluste...@127.0.0.1:8123) -> myself]
[INFO] [02/02/2017 10:33:16.841] [ClusterSystem-akka.actor.default-dispatcher-26] [akka.tcp://Cluste...@127.0.0.1:6123/system/sharding/shipmentCoordinator] Younger observed OldestChanged: [Some(akka.tcp://Cluste...@127.0.0.1:8123) -> myself]
[INFO] [02/02/2017 10:33:16.841] [ClusterSystem-akka.actor.default-dispatcher-3] [akka.tcp://Cluste...@127.0.0.1:6123/system/sharding/followerCoordinator] Younger observed OldestChanged: [Some(akka.tcp://Cluste...@127.0.0.1:8123) -> myself]
[INFO] [02/02/2017 10:33:16.841] [ClusterSystem-akka.actor.default-dispatcher-15] [akka.tcp://Cluste...@127.0.0.1:6123/system/sharding/siteCoordinator] Younger observed OldestChanged: [Some(akka.tcp://Cluste...@127.0.0.1:8123) -> myself]
[INFO] [02/02/2017 10:33:16.842] [ClusterSystem-akka.actor.default-dispatcher-29] [akka.tcp://Cluste...@127.0.0.1:6123/system/sharding/masterCoordinator] Younger observed OldestChanged: [Some(akka.tcp://Cluste...@127.0.0.1:8123) -> myself]
[INFO] [02/02/2017 10:33:16.842] [ClusterSystem-akka.actor.default-dispatcher-36] [akka.tcp://Cluste...@127.0.0.1:6123/system/sharding/locationCoordinator] ClusterSingletonManager state change [Younger -> BecomingOldest]
[INFO] [02/02/2017 10:33:16.842] [ClusterSystem-akka.actor.default-dispatcher-29] [akka.tcp://Cluste...@127.0.0.1:6123/system/sharding/masterCoordinator] ClusterSingletonManager state change [Younger -> BecomingOldest]
[INFO] [02/02/2017 10:33:16.842] [ClusterSystem-akka.actor.default-dispatcher-28] [akka.tcp://Cluste...@127.0.0.1:6123/system/sharding/eventCoordinator] ClusterSingletonManager state change [Younger -> BecomingOldest]
[INFO] [02/02/2017 10:33:16.842] [ClusterSystem-akka.actor.default-dispatcher-23] [akka.tcp://Cluste...@127.0.0.1:6123/system/sharding/shipment-issue-relationCoordinator] ClusterSingletonManager state change [Younger -> BecomingOldest]
[INFO] [02/02/2017 10:33:16.842] [ClusterSystem-akka.actor.default-dispatcher-26] [akka.tcp://Cluste...@127.0.0.1:6123/system/sharding/shipmentCoordinator] ClusterSingletonManager state change [Younger -> BecomingOldest]
[INFO] [02/02/2017 10:33:16.842] [ClusterSystem-akka.actor.default-dispatcher-15] [akka.tcp://Cluste...@127.0.0.1:6123/system/sharding/siteCoordinator] ClusterSingletonManager state change [Younger -> BecomingOldest]
[INFO] [02/02/2017 10:33:16.842] [ClusterSystem-akka.actor.default-dispatcher-21] [akka.tcp://Cluste...@127.0.0.1:6123/system/sharding/issueCoordinator] ClusterSingletonManager state change [Younger -> BecomingOldest]
[INFO] [02/02/2017 10:33:16.842] [ClusterSystem-akka.actor.default-dispatcher-3] [akka.tcp://Cluste...@127.0.0.1:6123/system/sharding/followerCoordinator] ClusterSingletonManager state change [Younger -> BecomingOldest]
[INFO] [02/02/2017 10:33:17.856] [ClusterSystem-akka.actor.default-dispatcher-4] [akka.tcp://Cluste...@127.0.0.1:6123/system/sharding/issueCoordinator] Retry [1], sending HandOverToMe to [Some(akka.tcp://Cluste...@127.0.0.1:8123)]
[INFO] [02/02/2017 10:33:17.856] [ClusterSystem-akka.actor.default-dispatcher-31] [akka.tcp://Cluste...@127.0.0.1:6123/system/sharding/masterCoordinator] Retry [1], sending HandOverToMe to [Some(akka.tcp://Cluste...@127.0.0.1:8123)]
[INFO] [02/02/2017 10:33:17.856] [ClusterSystem-akka.actor.default-dispatcher-29] [akka.tcp://Cluste...@127.0.0.1:6123/system/sharding/locationCoordinator] Retry [1], sending HandOverToMe to [Some(akka.tcp://Cluste...@127.0.0.1:8123)]
[INFO] [02/02/2017 10:33:17.856] [ClusterSystem-akka.actor.default-dispatcher-33] [akka.tcp://Cluste...@127.0.0.1:6123/system/sharding/shipment-issue-relationCoordinator] Retry [1], sending HandOverToMe to [Some(akka.tcp://Cluste...@127.0.0.1:8123)]
[INFO] [02/02/2017 10:33:17.856] [ClusterSystem-akka.actor.default-dispatcher-23] [akka.tcp://Cluste...@127.0.0.1:6123/system/sharding/shipmentCoordinator] Retry [1], sending HandOverToMe to [Some(akka.tcp://Cluste...@127.0.0.1:8123)]
[INFO] [02/02/2017 10:33:17.857] [ClusterSystem-akka.actor.default-dispatcher-3] [akka.tcp://Cluste...@127.0.0.1:6123/system/sharding/eventCoordinator] Retry [1], sending HandOverToMe to [Some(akka.tcp://Cluste...@127.0.0.1:8123)]
[INFO] [02/02/2017 10:33:17.857] [ClusterSystem-akka.actor.default-dispatcher-28] [akka.tcp://Cluste...@127.0.0.1:6123/system/sharding/followerCoordinator] Retry [1], sending HandOverToMe to [Some(akka.tcp://Cluste...@127.0.0.1:8123)]
[INFO] [02/02/2017 10:33:17.857] [ClusterSystem-akka.actor.default-dispatcher-20] [akka.tcp://Cluste...@127.0.0.1:6123/system/sharding/siteCoordinator] Retry [1], sending HandOverToMe to [Some(akka.tcp://Cluste...@127.0.0.1:8123)]
[INFO] [02/02/2017 10:33:18.836] [ClusterSystem-akka.actor.default-dispatcher-23] [akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://Cluste...@127.0.0.1:6123] - Marking exiting node(s) as UNREACHABLE [Member(address = akka.tcp://Cluste...@127.0.0.1:8123, status = Exiting)]. This is expected and they will be removed.
[INFO] [02/02/2017 10:33:18.874] [ClusterSystem-akka.actor.default-dispatcher-31] [akka.tcp://Cluste...@127.0.0.1:6123/system/sharding/issueCoordinator] Retry [2], sending HandOverToMe to [Some(akka.tcp://Cluste...@127.0.0.1:8123)]
[INFO] [02/02/2017 10:33:18.874] [ClusterSystem-akka.actor.default-dispatcher-23] [akka.tcp://Cluste...@127.0.0.1:6123/system/sharding/siteCoordinator] Retry [2], sending HandOverToMe to [Some(akka.tcp://Cluste...@127.0.0.1:8123)]
[INFO] [02/02/2017 10:33:18.874] [ClusterSystem-akka.actor.default-dispatcher-25] [akka.tcp://Cluste...@127.0.0.1:6123/system/sharding/followerCoordinator] Retry [2], sending HandOverToMe to [Some(akka.tcp://Cluste...@127.0.0.1:8123)]
[INFO] [02/02/2017 10:33:18.874] [ClusterSystem-akka.actor.default-dispatcher-35] [akka.tcp://Cluste...@127.0.0.1:6123/system/sharding/locationCoordinator] Retry [2], sending HandOverToMe to [Some(akka.tcp://Cluste...@127.0.0.1:8123)]
[INFO] [02/02/2017 10:33:18.874] [ClusterSystem-akka.actor.default-dispatcher-29] [akka.tcp://Cluste...@127.0.0.1:6123/system/sharding/shipment-issue-relationCoordinator] Retry [2], sending HandOverToMe to [Some(akka.tcp://Cluste...@127.0.0.1:8123)]
[INFO] [02/02/2017 10:33:18.874] [ClusterSystem-akka.actor.default-dispatcher-28] [akka.tcp://Cluste...@127.0.0.1:6123/system/sharding/eventCoordinator] Retry [2], sending HandOverToMe to [Some(akka.tcp://Cluste...@127.0.0.1:8123)]
[INFO] [02/02/2017 10:33:18.874] [ClusterSystem-akka.actor.default-dispatcher-20] [akka.tcp://Cluste...@127.0.0.1:6123/system/sharding/shipmentCoordinator] Retry [2], sending HandOverToMe to [Some(akka.tcp://Cluste...@127.0.0.1:8123)]
[INFO] [02/02/2017 10:33:18.874] [ClusterSystem-akka.actor.default-dispatcher-35] [akka.tcp://Cluste...@127.0.0.1:6123/system/sharding/masterCoordinator] Retry [2], sending HandOverToMe to [Some(akka.tcp://Cluste...@127.0.0.1:8123)]
[INFO] [02/02/2017 10:33:19.836] [ClusterSystem-akka.actor.default-dispatcher-21] [akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://Cluste...@127.0.0.1:6123] - Leader is removing exiting node [akka.tcp://Cluste...@127.0.0.1:8123]
[INFO] [02/02/2017 10:33:19.837] [ClusterSystem-akka.actor.default-dispatcher-17] [akka.tcp://Cluste...@127.0.0.1:6123/system/sharding/followerCoordinator] Previous oldest [akka.tcp://Cluste...@127.0.0.1:8123] removed
[INFO] [02/02/2017 10:33:19.837] [ClusterSystem-akka.actor.default-dispatcher-26] [akka.tcp://Cluste...@127.0.0.1:6123/system/sharding/issueCoordinator] Previous oldest [akka.tcp://Cluste...@127.0.0.1:8123] removed
[INFO] [02/02/2017 10:33:19.837] [ClusterSystem-akka.actor.default-dispatcher-19] [akka.tcp://Cluste...@127.0.0.1:6123/system/sharding/eventCoordinator] Previous oldest [akka.tcp://Cluste...@127.0.0.1:8123] removed
[INFO] [02/02/2017 10:33:19.837] [ClusterSystem-akka.actor.default-dispatcher-15] [akka.tcp://Cluste...@127.0.0.1:6123/system/sharding/shipment-issue-relationCoordinator] Previous oldest [akka.tcp://Cluste...@127.0.0.1:8123] removed
[INFO] [02/02/2017 10:33:19.837] [ClusterSystem-akka.actor.default-dispatcher-36] [akka.tcp://Cluste...@127.0.0.1:6123/system/sharding/locationCoordinator] Previous oldest [akka.tcp://Cluste...@127.0.0.1:8123] removed
[INFO] [02/02/2017 10:33:19.837] [ClusterSystem-akka.actor.default-dispatcher-23] [akka.tcp://Cluste...@127.0.0.1:6123/system/sharding/masterCoordinator] Previous oldest [akka.tcp://Cluste...@127.0.0.1:8123] removed
[INFO] [02/02/2017 10:33:19.837] [ClusterSystem-akka.actor.default-dispatcher-3] [akka.tcp://Cluste...@127.0.0.1:6123/system/sharding/siteCoordinator] Previous oldest [akka.tcp://Cluste...@127.0.0.1:8123] removed
[INFO] [02/02/2017 10:33:19.837] [ClusterSystem-akka.actor.default-dispatcher-17] [akka.tcp://Cluste...@127.0.0.1:6123/system/sharding/followerCoordinator] Singleton manager starting singleton actor [akka://ClusterSystem/system/sharding/followerCoordinator/singleton]
[INFO] [02/02/2017 10:33:19.837] [ClusterSystem-akka.actor.default-dispatcher-23] [akka.tcp://Cluste...@127.0.0.1:6123/system/sharding/masterCoordinator] Singleton manager starting singleton actor [akka://ClusterSystem/system/sharding/masterCoordinator/singleton]
[INFO] [02/02/2017 10:33:19.837] [ClusterSystem-akka.actor.default-dispatcher-3] [akka.tcp://Cluste...@127.0.0.1:6123/system/sharding/siteCoordinator] Singleton manager starting singleton actor [akka://ClusterSystem/system/sharding/siteCoordinator/singleton]
[INFO] [02/02/2017 10:33:19.838] [ClusterSystem-akka.actor.default-dispatcher-20] [akka.tcp://Cluste...@127.0.0.1:6123/system/sharding/shipmentCoordinator] Previous oldest [akka.tcp://Cluste...@127.0.0.1:8123] removed
[INFO] [02/02/2017 10:33:19.838] [ClusterSystem-akka.actor.default-dispatcher-36] [akka.tcp://Cluste...@127.0.0.1:6123/system/sharding/locationCoordinator] Singleton manager starting singleton actor [akka://ClusterSystem/system/sharding/locationCoordinator/singleton]
[INFO] [02/02/2017 10:33:19.838] [ClusterSystem-akka.actor.default-dispatcher-19] [akka.tcp://Cluste...@127.0.0.1:6123/system/sharding/eventCoordinator] Singleton manager starting singleton actor [akka://ClusterSystem/system/sharding/eventCoordinator/singleton]
[INFO] [02/02/2017 10:33:19.838] [ClusterSystem-akka.actor.default-dispatcher-20] [akka.tcp://Cluste...@127.0.0.1:6123/system/sharding/shipmentCoordinator] Singleton manager starting singleton actor [akka://ClusterSystem/system/sharding/shipmentCoordinator/singleton]
[INFO] [02/02/2017 10:33:19.838] [ClusterSystem-akka.actor.default-dispatcher-15] [akka.tcp://Cluste...@127.0.0.1:6123/system/sharding/shipment-issue-relationCoordinator] Singleton manager starting singleton actor [akka://ClusterSystem/system/sharding/shipment-issue-relationCoordinator/singleton]
[INFO] [02/02/2017 10:33:19.838] [ClusterSystem-akka.actor.default-dispatcher-26] [akka.tcp://Cluste...@127.0.0.1:6123/system/sharding/issueCoordinator] Singleton manager starting singleton actor [akka://ClusterSystem/system/sharding/issueCoordinator/singleton]
[INFO] [02/02/2017 10:33:19.838] [ClusterSystem-akka.actor.default-dispatcher-15] [akka.tcp://Cluste...@127.0.0.1:6123/system/sharding/shipment-issue-relationCoordinator] ClusterSingletonManager state change [BecomingOldest -> Oldest]
[INFO] [02/02/2017 10:33:19.838] [ClusterSystem-akka.actor.default-dispatcher-20] [akka.tcp://Cluste...@127.0.0.1:6123/system/sharding/shipmentCoordinator] ClusterSingletonManager state change [BecomingOldest -> Oldest]
[INFO] [02/02/2017 10:33:19.838] [ClusterSystem-akka.actor.default-dispatcher-26] [akka.tcp://Cluste...@127.0.0.1:6123/system/sharding/issueCoordinator] ClusterSingletonManager state change [BecomingOldest -> Oldest]
[INFO] [02/02/2017 10:33:19.838] [ClusterSystem-akka.actor.default-dispatcher-17] [akka.tcp://Cluste...@127.0.0.1:6123/system/sharding/followerCoordinator] ClusterSingletonManager state change [BecomingOldest -> Oldest]
[INFO] [02/02/2017 10:33:19.839] [ClusterSystem-akka.actor.default-dispatcher-19] [akka.tcp://Cluste...@127.0.0.1:6123/system/sharding/eventCoordinator] ClusterSingletonManager state change [BecomingOldest -> Oldest]
[INFO] [02/02/2017 10:33:19.839] [ClusterSystem-akka.actor.default-dispatcher-36] [akka.tcp://Cluste...@127.0.0.1:6123/system/sharding/locationCoordinator] ClusterSingletonManager state change [BecomingOldest -> Oldest]
[INFO] [02/02/2017 10:33:19.839] [ClusterSystem-akka.actor.default-dispatcher-23] [akka.tcp://Cluste...@127.0.0.1:6123/system/sharding/masterCoordinator] ClusterSingletonManager state change [BecomingOldest -> Oldest]
[INFO] [02/02/2017 10:33:19.839] [ClusterSystem-akka.actor.default-dispatcher-3] [akka.tcp://Cluste...@127.0.0.1:6123/system/sharding/siteCoordinator] ClusterSingletonManager state change [BecomingOldest -> Oldest]
[INFO] [02/02/2017 10:33:19.854] [ClusterSystem-akka.actor.default-dispatcher-17] [akka.persistence.Persistence(akka://ClusterSystem)] Auto-starting snapshot store `jdbc-snapshot-store`
2017-02-02T18:33:19,905Z [ClusterSystem-akka.persistence.dispatchers.default-plugin-dispatcher-37] WARN   com.zaxxer.hikari.HikariConfig.setJdbc4ConnectionTest 436  - The jdbcConnectionTest property is now deprecated, see the documentation for connectionTestQuery
2017-02-02T18:33:19,905Z [ClusterSystem-akka.persistence.dispatchers.default-plugin-dispatcher-38] WARN   com.zaxxer.hikari.HikariConfig.setJdbc4ConnectionTest 436  - The jdbcConnectionTest property is now deprecated, see the documentation for connectionTestQuery
2017-02-02T18:33:19,909Z [ClusterSystem-akka.persistence.dispatchers.default-plugin-dispatcher-38] INFO   com.zaxxer.hikari.HikariDataSource.<init> 70  - slick.db - Started.
2017-02-02T18:33:19,909Z [ClusterSystem-akka.persistence.dispatchers.default-plugin-dispatcher-37] INFO   com.zaxxer.hikari.HikariDataSource.<init> 70  - slick.db - Started.