ReliableDeliverySupervisor trying to connect node that is no longer in cluster

[Marek Żebrowski]

W have a setup in which some nodes are auto-scaled 

Even after clean node exit (DOWN)  other nodes tries to communicate with already left node:

WARN  a.r.ReliableDeliverySupervisor: Association with remote system [akka.tcp://sgAc...@app-2016-01-31-224114.as.sgrouples.com:2552] has failed, address is now gated for [100] ms. Reason: [Association failed with [akka.tcp://sgAc...@app-2016-01-31-224114.as.sgrouples.com:2552]] Caused by: [No response from remote for outbound association. Associate timed out after [15000 ms].]

that node is not in `ureachable` state - it does not exist in a cluster at all.

Where to look for a cause of such issues ?

[Patrik Nordwall]

What version are you using? Are you using Cluster Sharding?

/Patrik

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--

Patrik Nordwall
Typesafe -  Reactive apps on the JVM
Twitter: @patriknw

[Marek Żebrowski]

Yes, I'm using cluster sharding with persistence

W dniu wtorek, 2 lutego 2016 13:18:43 UTC+1 użytkownik Patrik Nordwall napisał:

What version are you using? Are you using Cluster Sharding?

/Patrik

On Tue, Feb 2, 2016 at 10:58 AM, Marek Żebrowski <marek.z...@gmail.com> wrote:

W have a setup in which some nodes are auto-scaled 

Even after clean node exit (DOWN)  other nodes tries to communicate with already left node:

WARN  a.r.ReliableDeliverySupervisor: Association with remote system [akka.tcp://sgActors@app-2016-01-31-224114.as.sgrouples.com:2552] has failed, address is now gated for [100] ms. Reason: [Association failed with [akka.tcp://sgActors@app-2016-01-31-224114.as.sgrouples.com:2552]] Caused by: [No response from remote for outbound association. Associate timed out after [15000 ms].]

that node is not in `ureachable` state - it does not exist in a cluster at all.

Where to look for a cause of such issues ?

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

[Patrik Nordwall]

On Tue, Feb 2, 2016 at 2:07 PM, Marek Żebrowski <marek.z...@gmail.com> wrote:

Yes, I'm using cluster sharding with persistence

The it is probably the sharding that triggers these connection attempts based on stored locations. That is harmless. I think we did some improvements of this and that is why I asked which version you are using.

 

W dniu wtorek, 2 lutego 2016 13:18:43 UTC+1 użytkownik Patrik Nordwall napisał:

What version are you using? Are you using Cluster Sharding?

/Patrik

On Tue, Feb 2, 2016 at 10:58 AM, Marek Żebrowski <marek.z...@gmail.com> wrote:

W have a setup in which some nodes are auto-scaled 

Even after clean node exit (DOWN)  other nodes tries to communicate with already left node:

WARN  a.r.ReliableDeliverySupervisor: Association with remote system [akka.tcp://sgAc...@app-2016-01-31-224114.as.sgrouples.com:2552] has failed, address is now gated for [100] ms. Reason: [Association failed with [akka.tcp://sgAc...@app-2016-01-31-224114.as.sgrouples.com:2552]] Caused by: [No response from remote for outbound association. Associate timed out after [15000 ms].]

[Marek Żebrowski]

Oh, sorry - I'm using akka 2.4.1 with https://github.com/scullxbones/akka-persistence-mongo persistence plugin.

Our nodes have downing procedure that does 

context.watch(region)
region ! ShardRegion.GracefulShutdown

and upon succesfull terminate we do cluster.leave ,wait for removal and stop jvm

You received this message because you are subscribed to a topic in the Google Groups "Akka User List" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/akka-user/ETwCwvKmWUU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to akka-user+...@googlegroups.com.

To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--

Marek Żebrowski

[Patrik Nordwall]

The sharding improvement that I was thinking about is in 2.4.0 (and 2.4.1), so then it is perhaps something else. Something is sending messages to it. You can enable more logging http://doc.akka.io/docs/akka/2.4.1/scala/logging.html#Auxiliary_remote_logging_options