PhiAccrualFailureDetector Constantly Spamming logs but nodes seem unaffected.

[kraythe]

We are constantly getting messages like this in our logs. 

2017-11-30 14:28:31,181 [WARN ] [PhiAccrualFailureDetector] (akka.remote.PhiAccrualFailureDetector@808f8ad) - heartbeat interval is growing too large: 1766 millis

I understand the theory behind this for early detection of node failure but despite that our nodes seem unaffected in response time. So I am wondering a couple of things. 

1) Why would the heartbeat be growing at all and how can it be improved? 

2) What is considered a bad heartbeat timing. 

Thanks a bunch. 

[Patrik Nordwall]

One reason could be that you are sending too large messages.

/Patrik

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

[Lukas Phaf]

I have noticed a lot of these as well, while they weren't occurring in the past. The message we are sending are pretty small.

I would have to dig to see if I can find when exactly these started occurring, but if I have to guess, I would say this started happening when we moved from Akka 2.4 tot 2.5.

Lukas

[Patrik Nordwall]

That log message is rather new. It's just a warning, indication that something might be unstable. Logs when the heartbeat interval exceeds half of the acceptable-heartbeat-pause. You can increase acceptable-heartbeat-pause to make it less sensitive.

--

>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.

To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscribe@googlegroups.com.

To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--

Patrik Nordwall
Akka Tech Lead
Lightbend -  Reactive apps on the JVM
Twitter: @patriknw

[kraythe]

Yeah I didn't see them before 2.5 either. What is weird is they happen when the cluster is doing almost nothing. 

-- Robert

[Alan Johnson]

Having just seen this message for the first time, my feedback would be that it's not at all clear what the message pertains to. I'm curious whether anyone's thought there might be some use in attaching reference codes to certain log messages so that it's easy to get context.

On Friday, December 1, 2017 at 3:24:28 AM UTC-5, Patrik Nordwall wrote:

That log message is rather new. It's just a warning, indication that something might be unstable. Logs when the heartbeat interval exceeds half of the acceptable-heartbeat-pause. You can increase acceptable-heartbeat-pause to make it less sensitive.

On Thu, Nov 30, 2017 at 8:18 PM, Lukas Phaf <lukas...@gmail.com> wrote:

I have noticed a lot of these as well, while they weren't occurring in the past. The message we are sending are pretty small.

I would have to dig to see if I can find when exactly these started occurring, but if I have to guess, I would say this started happening when we moved from Akka 2.4 tot 2.5.

Lukas

On Thursday, 30 November 2017 15:31:58 UTC+1, kraythe wrote:

We are constantly getting messages like this in our logs. 

2017-11-30 14:28:31,181 [WARN ] [PhiAccrualFailureDetector] (akka.remote.PhiAccrualFailureDetector@808f8ad) - heartbeat interval is growing too large: 1766 millis

I understand the theory behind this for early detection of node failure but despite that our nodes seem unaffected in response time. So I am wondering a couple of things. 

1) Why would the heartbeat be growing at all and how can it be improved? 

2) What is considered a bad heartbeat timing. 

Thanks a bunch. 

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.

To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.

To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.