[Cluster singleton] Trying to identify singleton

[Anders Båtstrand]

Dear group

We are using Akka 2.3.14 (unable to use Java 8 for now), and sometime when we do a rolling restart in our cluster, the singleton actor is not started, and we see the following in our logs.

I am not sure what is supposed to happen now, and how I can debug it any further. The cluster seems fine, all members know of each other and agree on a leader. The only thing missing is that the singleton actor is started.

Best regards,

Anders Båtstrand

2015-12-15 10:26:58,525 DEBUG akka.contrib.pattern.ClusterSingletonProxy my-actor-system-akka.actor.default-dispatcher-18 - Trying to identify singleton at akka.tcp://my-actor-system@hostname1:15553/user/MySingletonActor-manager/MySingletonActor
2015-12-15 10:26:58,018 DEBUG akka.contrib.pattern.ClusterSingletonProxy my-actor-system-akka.actor.default-dispatcher-2 - Trying to identify singleton at akka.tcp://my-actor-system@hostname1:15553/user/MySingletonActor-manager/MySingletonActor
[...]
2015-12-15 10:26:52,009 DEBUG akka.contrib.pattern.ClusterSingletonProxy my-actor-system-akka.actor.default-dispatcher-4 - Trying to identify singleton at akka.tcp://my-actor-system@hostname1:15553/user/MySingletonActor-manager/MySingletonActor
2015-12-15 10:26:51,528 DEBUG akka.contrib.pattern.ClusterSingletonProxy my-actor-system-akka.actor.default-dispatcher-19 - Trying to identify singleton at akka.tcp://my-actor-system@hostname1:15553/user/MySingletonActor-manager/MySingletonActor
2015-12-15 10:26:51,003 DEBUG akka.contrib.pattern.ClusterSingletonProxy my-actor-system-akka.actor.default-dispatcher-19 - Trying to identify singleton at akka.tcp://my-actor-system@hostname1:15553/user/MySingletonActor-manager/MySingletonActor
2015-12-15 10:26:50,756 DEBUG akka.contrib.pattern.ClusterSingletonProxy my-actor-system-akka.actor.default-dispatcher-14 - No singleton available, stashing message my.Message@50b0ea8b
2015-12-15 10:26:50,729 DEBUG akka.contrib.pattern.ClusterSingletonProxy my-actor-system-akka.actor.default-dispatcher-16 - Forwarding message to current singleton instance my.Message@4ada4efd
2015-12-15 10:26:50,527 DEBUG akka.contrib.pattern.ClusterSingletonProxy my-actor-system-akka.actor.default-dispatcher-3 - Trying to identify singleton at akka.tcp://my-actor-system@hostname1:15553/user/MySingletonActor-manager/MySingletonActor
2015-12-15 10:26:50,015 DEBUG akka.contrib.pattern.ClusterSingletonProxy my-actor-system-akka.actor.default-dispatcher-17 - Trying to identify singleton at akka.tcp://my-actor-system@hostname1:15553/user/MySingletonActor-manager/MySingletonActor
2015-12-15 10:26:49,669 DEBUG akka.contrib.pattern.ClusterSingletonProxy my-actor-system-akka.actor.default-dispatcher-4 - No singleton available, stashing message my.Message@2ff5aa1 2015-12-15 10:26:49,658 DEBUG akka.contrib.pattern.ClusterSingletonProxy my-actor-system-akka.actor.default-dispatcher-16 - Forwarding message to current singleton instance my.Message@d62aa707

[Patrik Nordwall]

Sounds strange. Were the members removed correctly after the rolling restart? No dangling unreachable?

/Patrik

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--

Patrik Nordwall
Typesafe -  Reactive apps on the JVM
Twitter: @patriknw

[Anders Båtstrand]

Thanks for your reply, Patrik!

We use a setup with auto-down after 10 seconds, so I think that is not
possible. I will do the test again, however, and watch out for any
warnings when the nodes goes down (leaves the cluster).

Anders

> You received this message because you are subscribed to a topic in the

> Google Groups "Akka User List" group.

> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/akka-user/EOGlUk6TVOU/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to

[Anders Båtstrand]

I tried this again, and looked in the logs. The nodes are downed as expected, but the following was interesting. Could it be related?

2015-12-22 13:34:39,983 DEBUG akka.cluster.ClusterCoreDaemon
my-system-akka.actor.default-dispatcher-15 - Couldn't establish a causal
relationship between "remote" gossip and "local" gossip -
Remote[Gossip(members = [Member(address = akka.tcp://my-system@hostname1:15553,
status = Exiting), Member(address = akka.tcp://my-system@hostname2:15553,
status = Up)], overview = GossipOverview(reachability = [], seen =
[UniqueAddress(akka.tcp://my-system@hostname1:15553,302214267)]), version =
VectorClock(0a39d7a0596e4c3eee3c50756da01570 -> 7,
0ca40c47022419c420b6324b277b52fc -> 5, 1458acfed85b45ff5cbed5be74982726
-> 7, 5678be7b6839a169562189f328228469 -> 8,
58c9531ffe15d1e940c85ee05108c948 -> 5, 684ff781d2a79f145a725b06e2af660b
-> 7, 836ef397098eaaaac3dbf910b45aa546 -> 5,
a80c50a2c77f97610b08e5c14c8e2817 -> 12, d27b32918e981aed9c2eecee85271046
-> 5, d296fbb49c216e0426c3294a764a4e84 -> 5,
f8abb31f0ddbb57f41e32b310561cb95 -> 7, fbc07be2fb2fc1a3e34b9fb4dffe3b80
-> 7, fc4e18d72ccc6089e42a05fbec1cf782 -> 7))] - Local[Gossip(members =
[Member(address = akka.tcp://my-system@hostname1:15553, status = Leaving),
Member(address = akka.tcp://my-system@hostname2:15553, status = Leaving)],
overview = GossipOverview(reachability = [], seen =
[UniqueAddress(akka.tcp://my-system@hostname2:15553,208253255)]), version =
VectorClock(0a39d7a0596e4c3eee3c50756da01570 -> 7,
0ca40c47022419c420b6324b277b52fc -> 5, 1458acfed85b45ff5cbed5be74982726
-> 7, 4f227b3e72e9f062b50cb511d0591eba -> 1,
5678be7b6839a169562189f328228469 -> 8, 58c9531ffe15d1e940c85ee05108c948
-> 5, 684ff781d2a79f145a725b06e2af660b -> 7,
836ef397098eaaaac3dbf910b45aa546 -> 5, a80c50a2c77f97610b08e5c14c8e2817
-> 12, d27b32918e981aed9c2eecee85271046 -> 5,
d296fbb49c216e0426c3294a764a4e84 -> 5, f8abb31f0ddbb57f41e32b310561cb95
-> 6, fbc07be2fb2fc1a3e34b9fb4dffe3b80 -> 7,
fc4e18d72ccc6089e42a05fbec1cf782 -> 7))] - merged them into [Gossip(members
= [Member(address = akka.tcp://my-system@hostname1:15553, status = Exiting),
Member(address = akka.tcp://my-system@hostname2:15553, status = Leaving)],
overview = GossipOverview(reachability = [], seen = []), version =
VectorClock(0a39d7a0596e4c3eee3c50756da01570 -> 7,
0ca40c47022419c420b6324b277b52fc -> 5, 1458acfed85b45ff5cbed5be74982726
-> 7, 4f227b3e72e9f062b50cb511d0591eba -> 1, 5678be7b6839a169562189f328228469
-> 8, 58c9531ffe15d1e940c85ee05108c948 -> 5,
684ff781d2a79f145a725b06e2af660b -> 7, 836ef397098eaaaac3dbf910b45aa546
-> 5, a80c50a2c77f97610b08e5c14c8e2817 -> 12,
d27b32918e981aed9c2eecee85271046 -> 5, d296fbb49c216e0426c3294a764a4e84
-> 5, f8abb31f0ddbb57f41e32b310561cb95 -> 7,
fbc07be2fb2fc1a3e34b9fb4dffe3b80 -> 7, fc4e18d72ccc6089e42a05fbec1cf782
-> 7))]

[Patrik Nordwall]

no, that is harmless and the merged result looks correct

--

>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.