Re: [akka-user] AKKA cluster heartbeat delay

[Patrik Nordwall]

It can be many reasons for delayed heartbeat messages.

- GC. Enable gc logging and try to correlate.
- Slow serialization/deserialization of your application messages, especially if you send many messages in a burst
- Running blocking actors/tasks on the default dispatcher

/Patrik

tors 25 maj 2017 kl. 13:58 skrev Yutao Shuai <szq...@gmail.com>:

We have a 3 nodes cluster （157,158 and 159）, we send 4000 messages per seconds to 158 in 157, sometimes 157 can not receive the heartbeat from 158, but 158 is already sended. The heartbeat between 159 and 158 is fine, and 157-159 also ok. Our version is akka2.3.10, and we want to solve this problem in 2.3.10, we will upgrade to 2.4.18 soon.

Here is the log of heartbeat, we add a log when receive heartbeat.

This is 157 log, as we can see, 157 can not receive the heartbeat from 158 since 20:59:15,985 until 20:59:32,558, Heartbeat is not lost but delay

2017-05-23 20:59:07,516 | DEBUG | ult-dispatcher-3 | ClusterHeartbeatReceiver         |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Received Heartbeat from akka.tcp://actor...@192.168.0.158:2551

2017-05-23 20:59:08,601 | DEBUG | ult-dispatcher-2 | ClusterHeartbeatReceiver         |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Received Heartbeat from akka.tcp://actor...@192.168.0.158:2551

2017-05-23 20:59:11,824 | DEBUG | t-dispatcher-852 | ClusterHeartbeatReceiver         |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Received Heartbeat from akka.tcp://actor...@192.168.0.158:2551

2017-05-23 20:59:13,004 | DEBUG | lt-dispatcher-29 | ClusterHeartbeatReceiver         |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Received Heartbeat from akka.tcp://actor...@192.168.0.158:2551

2017-05-23 20:59:13,166 | DEBUG | t-dispatcher-853 | ClusterHeartbeatReceiver         |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Received Heartbeat from akka.tcp://actor...@192.168.0.158:2551

2017-05-23 20:59:13,273 | DEBUG | t-dispatcher-853 | ClusterHeartbeatReceiver         |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Received Heartbeat from akka.tcp://actor...@192.168.0.158:2551

2017-05-23 20:59:14,287 | DEBUG | t-dispatcher-853 | ClusterHeartbeatReceiver         |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Received Heartbeat from akka.tcp://actor...@192.168.0.158:2551

2017-05-23 20:59:15,853 | DEBUG | t-dispatcher-853 | ClusterHeartbeatReceiver         |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Received Heartbeat from akka.tcp://actor...@192.168.0.158:2551

2017-05-23 20:59:15,985 | DEBUG | t-dispatcher-852 | ClusterHeartbeatReceiver         |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Received Heartbeat from akka.tcp://actor...@192.168.0.158:2551

2017-05-23 20:59:32,558 | DEBUG | ult-dispatcher-3 | ClusterHeartbeatReceiver         |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Received Heartbeat from akka.tcp://actor...@192.168.0.158:2551

2017-05-23 20:59:32,612 | DEBUG | ult-dispatcher-3 | ClusterHeartbeatReceiver         |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Received Heartbeat from akka.tcp://actor...@192.168.0.158:2551

2017-05-23 20:59:32,632 | DEBUG | ult-dispatcher-3 | ClusterHeartbeatReceiver         |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Received Heartbeat from akka.tcp://actor...@192.168.0.158:2551

2017-05-23 20:59:32,831 | DEBUG | ult-dispatcher-3 | ClusterHeartbeatReceiver         |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Received Heartbeat from akka.tcp://actor...@192.168.0.158:2551

2017-05-23 20:59:32,910 | DEBUG | ult-dispatcher-3 | ClusterHeartbeatReceiver         |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Received Heartbeat from akka.tcp://actor...@192.168.0.158:2551

2017-05-23 20:59:32,982 | DEBUG | ult-dispatcher-3 | ClusterHeartbeatReceiver         |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Received Heartbeat from akka.tcp://actor...@192.168.0.158:2551

2017-05-23 20:59:32,987 | DEBUG | ult-dispatcher-3 | ClusterHeartbeatReceiver         |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Received Heartbeat from akka.tcp://actor...@192.168.0.158:2551

2017-05-23 20:59:33,059 | DEBUG | ult-dispatcher-2 | ClusterHeartbeatReceiver         |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Received Heartbeat from akka.tcp://actor...@192.168.0.158:2551

2017-05-23 20:59:33,101 | DEBUG | ult-dispatcher-2 | ClusterHeartbeatReceiver         |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Received Heartbeat from akka.tcp://actor...@192.168.0.158:2551

2017-05-23 20:59:33,170 | DEBUG | ult-dispatcher-2 | ClusterHeartbeatReceiver         |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Received Heartbeat from akka.tcp://actor...@192.168.0.158:2551

2017-05-23 20:59:33,175 | DEBUG | ult-dispatcher-2 | ClusterHeartbeatReceiver         |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Received Heartbeat from akka.tcp://actor...@192.168.0.158:2551

2017-05-23 20:59:33,190 | DEBUG | ult-dispatcher-2 | ClusterHeartbeatReceiver         |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Received Heartbeat from akka.tcp://actor...@192.168.0.158:2551

2017-05-23 20:59:33,224 | DEBUG | ult-dispatcher-2 | ClusterHeartbeatReceiver         |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Received Heartbeat from akka.tcp://actor...@192.168.0.158:2551

2017-05-23 20:59:33,226 | DEBUG | ult-dispatcher-2 | ClusterHeartbeatReceiver         |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Received Heartbeat from akka.tcp://actor...@192.168.0.158:2551

2017-05-23 20:59:33,269 | DEBUG | ult-dispatcher-2 | ClusterHeartbeatReceiver         |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Received Heartbeat from akka.tcp://actor...@192.168.0.158:2551

2017-05-23 20:59:33,302 | DEBUG | ult-dispatcher-2 | ClusterHeartbeatReceiver         |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Received Heartbeat from akka.tcp://actor...@192.168.0.158:2551

2017-05-23 20:59:33,348 | DEBUG | ult-dispatcher-2 | ClusterHeartbeatReceiver         |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Received Heartbeat from akka.tcp://actor...@192.168.0.158:2551

2017-05-23 20:59:33,369 | DEBUG | ult-dispatcher-2 | ClusterHeartbeatReceiver         |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Received Heartbeat from akka.tcp://actor...@192.168.0.158:2551

2017-05-23 20:59:34,221 | DEBUG | ult-dispatcher-3 | ClusterHeartbeatReceiver         |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Received Heartbeat from akka.tcp://actor...@192.168.0.158:2551

2017-05-23 20:59:34,603 | DEBUG | ult-dispatcher-2 | ClusterHeartbeatReceiver         |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Received Heartbeat from akka.tcp://actor...@192.168.0.158:2551

Here is 158 log,  We can confirm that the heartbeat has been sent successfully

2017-05-23 20:59:15,265 | DEBUG | -dispatcher-2409 | ClusterHeartbeatSender           |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node [akka.tcp://actor...@192.168.0.158:2551] - Heartbeat to [akka.tcp://actor...@192.168.0.157:2551]

2017-05-23 20:59:16,266 | DEBUG | -dispatcher-2398 | ClusterHeartbeatSender           |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node [akka.tcp://actor...@192.168.0.158:2551] - Heartbeat to [akka.tcp://actor...@192.168.0.157:2551]

2017-05-23 20:59:17,266 | DEBUG | -dispatcher-2409 | ClusterHeartbeatSender           |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node [akka.tcp://actor...@192.168.0.158:2551] - Heartbeat to [akka.tcp://actor...@192.168.0.157:2551]

2017-05-23 20:59:18,265 | DEBUG | -dispatcher-2398 | ClusterHeartbeatSender           |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node [akka.tcp://actor...@192.168.0.158:2551] - Heartbeat to [akka.tcp://actor...@192.168.0.157:2551]

2017-05-23 20:59:19,265 | DEBUG | ult-dispatcher-3 | ClusterHeartbeatSender           |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node [akka.tcp://actor...@192.168.0.158:2551] - Heartbeat to [akka.tcp://actor...@192.168.0.157:2551]

2017-05-23 20:59:20,265 | DEBUG | ult-dispatcher-3 | ClusterHeartbeatSender           |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node [akka.tcp://actor...@192.168.0.158:2551] - Heartbeat to [akka.tcp://actor...@192.168.0.157:2551]

2017-05-23 20:59:21,266 | DEBUG | -dispatcher-2398 | ClusterHeartbeatSender           |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node [akka.tcp://actor...@192.168.0.158:2551] - Heartbeat to [akka.tcp://actor...@192.168.0.157:2551]

2017-05-23 20:59:22,287 | DEBUG | -dispatcher-2401 | ClusterHeartbeatSender           |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node [akka.tcp://actor...@192.168.0.158:2551] - Heartbeat to [akka.tcp://actor...@192.168.0.157:2551]

2017-05-23 20:59:23,265 | DEBUG | -dispatcher-2398 | ClusterHeartbeatSender           |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node [akka.tcp://actor...@192.168.0.158:2551] - Heartbeat to [akka.tcp://actor...@192.168.0.157:2551]

2017-05-23 20:59:24,265 | DEBUG | -dispatcher-2401 | ClusterHeartbeatSender           |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node [akka.tcp://actor...@192.168.0.158:2551] - Heartbeat to [akka.tcp://actor...@192.168.0.157:2551]

2017-05-23 20:59:25,265 | DEBUG | -dispatcher-2409 | ClusterHeartbeatSender           |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node [akka.tcp://actor...@192.168.0.158:2551] - Heartbeat to [akka.tcp://actor...@192.168.0.157:2551]

2017-05-23 20:59:26,266 | DEBUG | -dispatcher-2398 | ClusterHeartbeatSender           |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node [akka.tcp://actor...@192.168.0.158:2551] - Heartbeat to [akka.tcp://actor...@192.168.0.157:2551]

2017-05-23 20:59:27,265 | DEBUG | -dispatcher-2401 | ClusterHeartbeatSender           |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node [akka.tcp://actor...@192.168.0.158:2551] - Heartbeat to [akka.tcp://actor...@192.168.0.157:2551]

2017-05-23 20:59:28,265 | DEBUG | -dispatcher-2398 | ClusterHeartbeatSender           |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node [akka.tcp://actor...@192.168.0.158:2551] - Heartbeat to [akka.tcp://actor...@192.168.0.157:2551]

2017-05-23 20:59:29,265 | DEBUG | -dispatcher-2409 | ClusterHeartbeatSender           |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node [akka.tcp://actor...@192.168.0.158:2551] - Heartbeat to [akka.tcp://actor...@192.168.0.157:2551]

2017-05-23 20:59:30,266 | DEBUG | -dispatcher-2398 | ClusterHeartbeatSender           |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node [akka.tcp://actor...@192.168.0.158:2551] - Heartbeat to [akka.tcp://actor...@192.168.0.157:2551]

2017-05-23 20:59:31,265 | DEBUG | -dispatcher-2409 | ClusterHeartbeatSender           |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node [akka.tcp://actor...@192.168.0.158:2551] - Heartbeat to [akka.tcp://actor...@192.168.0.157:2551]

2017-05-23 20:59:32,265 | DEBUG | -dispatcher-2409 | ClusterHeartbeatSender           |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node [akka.tcp://actor...@192.168.0.158:2551] - Heartbeat to [akka.tcp://actor...@192.168.0.157:2551]

2017-05-23 20:59:33,266 | DEBUG | -dispatcher-2401 | ClusterHeartbeatSender           |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node [akka.tcp://actor...@192.168.0.158:2551] - Heartbeat to [akka.tcp://actor...@192.168.0.157:2551]

2017-05-23 20:59:34,265 | DEBUG | -dispatcher-2401 | ClusterHeartbeatSender           |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node [akka.tcp://actor...@192.168.0.158:2551] - Heartbeat to [akka.tcp://actor...@192.168.0.157:2551]

2017-05-23 20:59:35,266 | DEBUG | -dispatcher-2405 | ClusterHeartbeatSender           |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node [akka.tcp://actor...@192.168.0.158:2551] - Heartbeat to [akka.tcp://actor...@192.168.0.157:2551]

2017-05-23 20:59:36,270 | DEBUG | -dispatcher-2401 | ClusterHeartbeatSender           |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node [akka.tcp://actor...@192.168.0.158:2551] - Heartbeat to [akka.tcp://actor...@192.168.0.157:2551]

2017-05-23 20:59:37,265 | DEBUG | -dispatcher-2401 | ClusterHeartbeatSender           |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node [akka.tcp://actor...@192.168.0.158:2551] - Heartbeat to [akka.tcp://actor...@192.168.0.157:2551]

2017-05-23 20:59:38,266 | DEBUG | -dispatcher-2408 | ClusterHeartbeatSender           |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node [akka.tcp://actor...@192.168.0.158:2551] - Heartbeat to [akka.tcp://actor...@192.168.0.157:2551]

2017-05-23 20:59:39,265 | DEBUG | -dispatcher-2408 | ClusterHeartbeatSender           |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node [akka.tcp://actor...@192.168.0.158:2551] - Heartbeat to [akka.tcp://actor...@192.168.0.157:2551]

2017-05-23 20:59:40,265 | DEBUG | -dispatcher-2405 | ClusterHeartbeatSender           |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node [akka.tcp://actor...@192.168.0.158:2551] - Heartbeat to [akka.tcp://actor...@192.168.0.157:2551]

2017-05-23 20:59:41,265 | DEBUG | -dispatcher-2405 | ClusterHeartbeatSender           |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node [akka.tcp://actor...@192.168.0.158:2551] - Heartbeat to [akka.tcp://actor...@192.168.0.157:2551]

2017-05-23 20:59:42,265 | DEBUG | -dispatcher-2408 | ClusterHeartbeatSender           |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node [akka.tcp://actor...@192.168.0.158:2551] - Heartbeat to [akka.tcp://actor...@192.168.0.157:2551]

2017-05-23 20:59:43,265 | DEBUG | -dispatcher-2401 | ClusterHeartbeatSender           |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node [akka.tcp://actor...@192.168.0.158:2551] - Heartbeat to [akka.tcp://actor...@192.168.0.157:2551]

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

[Konstantinos Chaitas]

Hello,

I am facing also the issues with the delayed heartbeats. It is very strange because it is happening even without load. I created a repo, which you can use to reproduce it. Link : https://github.com/CostasChaitas/Akka-Demo

Any help would be much appreciated. Th

On Sunday, 28 May 2017 12:32:25 UTC+2, Patrik Nordwall wrote:

It can be many reasons for delayed heartbeat messages.

- GC. Enable gc logging and try to correlate.
- Slow serialization/deserialization of your application messages, especially if you send many messages in a burst
- Running blocking actors/tasks on the default dispatcher

/Patrik

tors 25 maj 2017 kl. 13:58 skrev Yutao Shuai <szq...@gmail.com>:

We have a 3 nodes cluster （157,158 and 159）, we send 4000 messages per seconds to 158 in 157, sometimes 157 can not receive the heartbeat from 158, but 158 is already sended. The heartbeat between 159 and 158 is fine, and 157-159 also ok. Our version is akka2.3.10, and we want to solve this problem in 2.3.10, we will upgrade to 2.4.18 soon.

Here is the log of heartbeat, we add a log when receive heartbeat.

This is 157 log, as we can see, 157 can not receive the heartbeat from 158 since 20:59:15,985 until 20:59:32,558, Heartbeat is not lost but delay

2017-05-23 20:59:07,516 | DEBUG | ult-dispatcher-3 | ClusterHeartbeatReceiver         |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Received Heartbeat from akka.tcp://actorsystem@192.168.0.158:2551

2017-05-23 20:59:08,601 | DEBUG | ult-dispatcher-2 | ClusterHeartbeatReceiver         |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Received Heartbeat from akka.tcp://actorsystem@192.168.0.158:2551

2017-05-23 20:59:11,824 | DEBUG | t-dispatcher-852 | ClusterHeartbeatReceiver         |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Received Heartbeat from akka.tcp://actorsystem@192.168.0.158:2551

2017-05-23 20:59:13,004 | DEBUG | lt-dispatcher-29 | ClusterHeartbeatReceiver         |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Received Heartbeat from akka.tcp://actorsystem@192.168.0.158:2551

2017-05-23 20:59:13,166 | DEBUG | t-dispatcher-853 | ClusterHeartbeatReceiver         |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Received Heartbeat from akka.tcp://actorsystem@192.168.0.158:2551

2017-05-23 20:59:13,273 | DEBUG | t-dispatcher-853 | ClusterHeartbeatReceiver         |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Received Heartbeat from akka.tcp://actorsystem@192.168.0.158:2551

2017-05-23 20:59:14,287 | DEBUG | t-dispatcher-853 | ClusterHeartbeatReceiver         |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Received Heartbeat from akka.tcp://actorsystem@192.168.0.158:2551

2017-05-23 20:59:15,853 | DEBUG | t-dispatcher-853 | ClusterHeartbeatReceiver         |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Received Heartbeat from akka.tcp://actorsystem@192.168.0.158:2551

2017-05-23 20:59:15,985 | DEBUG | t-dispatcher-852 | ClusterHeartbeatReceiver         |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Received Heartbeat from akka.tcp://actorsystem@192.168.0.158:2551

2017-05-23 20:59:32,558 | DEBUG | ult-dispatcher-3 | ClusterHeartbeatReceiver         |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Received Heartbeat from akka.tcp://actorsystem@192.168.0.158:2551

2017-05-23 20:59:32,612 | DEBUG | ult-dispatcher-3 | ClusterHeartbeatReceiver         |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Received Heartbeat from akka.tcp://actorsystem@192.168.0.158:2551

2017-05-23 20:59:32,632 | DEBUG | ult-dispatcher-3 | ClusterHeartbeatReceiver         |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Received Heartbeat from akka.tcp://actorsystem@192.168.0.158:2551

2017-05-23 20:59:32,831 | DEBUG | ult-dispatcher-3 | ClusterHeartbeatReceiver         |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Received Heartbeat from akka.tcp://actorsystem@192.168.0.158:2551

2017-05-23 20:59:32,910 | DEBUG | ult-dispatcher-3 | ClusterHeartbeatReceiver         |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Received Heartbeat from akka.tcp://actorsystem@192.168.0.158:2551

2017-05-23 20:59:32,982 | DEBUG | ult-dispatcher-3 | ClusterHeartbeatReceiver         |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Received Heartbeat from akka.tcp://actorsystem@192.168.0.158:2551

2017-05-23 20:59:32,987 | DEBUG | ult-dispatcher-3 | ClusterHeartbeatReceiver         |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Received Heartbeat from akka.tcp://actorsystem@192.168.0.158:2551

2017-05-23 20:59:33,059 | DEBUG | ult-dispatcher-2 | ClusterHeartbeatReceiver         |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Received Heartbeat from akka.tcp://actorsystem@192.168.0.158:2551

2017-05-23 20:59:33,101 | DEBUG | ult-dispatcher-2 | ClusterHeartbeatReceiver         |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Received Heartbeat from akka.tcp://actorsystem@192.168.0.158:2551

2017-05-23 20:59:33,170 | DEBUG | ult-dispatcher-2 | ClusterHeartbeatReceiver         |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Received Heartbeat from akka.tcp://actorsystem@192.168.0.158:2551

2017-05-23 20:59:33,175 | DEBUG | ult-dispatcher-2 | ClusterHeartbeatReceiver         |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Received Heartbeat from akka.tcp://actorsystem@192.168.0.158:2551

2017-05-23 20:59:33,190 | DEBUG | ult-dispatcher-2 | ClusterHeartbeatReceiver         |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Received Heartbeat from akka.tcp://actorsystem@192.168.0.158:2551

2017-05-23 20:59:33,224 | DEBUG | ult-dispatcher-2 | ClusterHeartbeatReceiver         |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Received Heartbeat from akka.tcp://actorsystem@192.168.0.158:2551

2017-05-23 20:59:33,226 | DEBUG | ult-dispatcher-2 | ClusterHeartbeatReceiver         |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Received Heartbeat from akka.tcp://actorsystem@192.168.0.158:2551

2017-05-23 20:59:33,269 | DEBUG | ult-dispatcher-2 | ClusterHeartbeatReceiver         |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Received Heartbeat from akka.tcp://actorsystem@192.168.0.158:2551

2017-05-23 20:59:33,302 | DEBUG | ult-dispatcher-2 | ClusterHeartbeatReceiver         |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Received Heartbeat from akka.tcp://actorsystem@192.168.0.158:2551

2017-05-23 20:59:33,348 | DEBUG | ult-dispatcher-2 | ClusterHeartbeatReceiver         |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Received Heartbeat from akka.tcp://actorsystem@192.168.0.158:2551

2017-05-23 20:59:33,369 | DEBUG | ult-dispatcher-2 | ClusterHeartbeatReceiver         |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Received Heartbeat from akka.tcp://actorsystem@192.168.0.158:2551

2017-05-23 20:59:34,221 | DEBUG | ult-dispatcher-3 | ClusterHeartbeatReceiver         |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Received Heartbeat from akka.tcp://actorsystem@192.168.0.158:2551

2017-05-23 20:59:34,603 | DEBUG | ult-dispatcher-2 | ClusterHeartbeatReceiver         |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Received Heartbeat from akka.tcp://actorsystem@192.168.0.158:2551

Here is 158 log,  We can confirm that the heartbeat has been sent successfully

2017-05-23 20:59:15,265 | DEBUG | -dispatcher-2409 | ClusterHeartbeatSender           |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node [akka.tcp://actorsystem@192.168.0.158:2551] - Heartbeat to [akka.tcp://actorsystem@192.168.0.157:2551]

2017-05-23 20:59:16,266 | DEBUG | -dispatcher-2398 | ClusterHeartbeatSender           |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node [akka.tcp://actorsystem@192.168.0.158:2551] - Heartbeat to [akka.tcp://actorsystem@192.168.0.157:2551]

2017-05-23 20:59:17,266 | DEBUG | -dispatcher-2409 | ClusterHeartbeatSender           |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node [akka.tcp://actorsystem@192.168.0.158:2551] - Heartbeat to [akka.tcp://actorsystem@192.168.0.157:2551]

2017-05-23 20:59:18,265 | DEBUG | -dispatcher-2398 | ClusterHeartbeatSender           |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node [akka.tcp://actorsystem@192.168.0.158:2551] - Heartbeat to [akka.tcp://actorsystem@192.168.0.157:2551]

2017-05-23 20:59:19,265 | DEBUG | ult-dispatcher-3 | ClusterHeartbeatSender           |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node [akka.tcp://actorsystem@192.168.0.158:2551] - Heartbeat to [akka.tcp://actorsystem@192.168.0.157:2551]

2017-05-23 20:59:20,265 | DEBUG | ult-dispatcher-3 | ClusterHeartbeatSender           |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node [akka.tcp://actorsystem@192.168.0.158:2551] - Heartbeat to [akka.tcp://actorsystem@192.168.0.157:2551]

2017-05-23 20:59:21,266 | DEBUG | -dispatcher-2398 | ClusterHeartbeatSender           |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node [akka.tcp://actorsystem@192.168.0.158:2551] - Heartbeat to [akka.tcp://actorsystem@192.168.0.157:2551]

2017-05-23 20:59:22,287 | DEBUG | -dispatcher-2401 | ClusterHeartbeatSender           |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node [akka.tcp://actorsystem@192.168.0.158:2551] - Heartbeat to [akka.tcp://actorsystem@192.168.0.157:2551]

2017-05-23 20:59:23,265 | DEBUG | -dispatcher-2398 | ClusterHeartbeatSender           |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node [akka.tcp://actorsystem@192.168.0.158:2551] - Heartbeat to [akka.tcp://actorsystem@192.168.0.157:2551]

2017-05-23 20:59:24,265 | DEBUG | -dispatcher-2401 | ClusterHeartbeatSender           |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node [akka.tcp://actorsystem@192.168.0.158:2551] - Heartbeat to [akka.tcp://actorsystem@192.168.0.157:2551]

2017-05-23 20:59:25,265 | DEBUG | -dispatcher-2409 | ClusterHeartbeatSender           |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node [akka.tcp://actorsystem@192.168.0.158:2551] - Heartbeat to [akka.tcp://actorsystem@192.168.0.157:2551]

2017-05-23 20:59:26,266 | DEBUG | -dispatcher-2398 | ClusterHeartbeatSender           |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node [akka.tcp://actorsystem@192.168.0.158:2551] - Heartbeat to [akka.tcp://actorsystem@192.168.0.157:2551]

2017-05-23 20:59:27,265 | DEBUG | -dispatcher-2401 | ClusterHeartbeatSender           |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node [akka.tcp://actorsystem@192.168.0.158:2551] - Heartbeat to [akka.tcp://actorsystem@192.168.0.157:2551]

2017-05-23 20:59:28,265 | DEBUG | -dispatcher-2398 | ClusterHeartbeatSender           |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node [akka.tcp://actorsystem@192.168.0.158:2551] - Heartbeat to [akka.tcp://actorsystem@192.168.0.157:2551]

2017-05-23 20:59:29,265 | DEBUG | -dispatcher-2409 | ClusterHeartbeatSender           |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node [akka.tcp://actorsystem@192.168.0.158:2551] - Heartbeat to [akka.tcp://actorsystem@192.168.0.157:2551]

2017-05-23 20:59:30,266 | DEBUG | -dispatcher-2398 | ClusterHeartbeatSender           |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node [akka.tcp://actorsystem@192.168.0.158:2551] - Heartbeat to [akka.tcp://actorsystem@192.168.0.157:2551]

2017-05-23 20:59:31,265 | DEBUG | -dispatcher-2409 | ClusterHeartbeatSender           |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node [akka.tcp://actorsystem@192.168.0.158:2551] - Heartbeat to [akka.tcp://actorsystem@192.168.0.157:2551]

2017-05-23 20:59:32,265 | DEBUG | -dispatcher-2409 | ClusterHeartbeatSender           |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node [akka.tcp://actorsystem@192.168.0.158:2551] - Heartbeat to [akka.tcp://actorsystem@192.168.0.157:2551]

2017-05-23 20:59:33,266 | DEBUG | -dispatcher-2401 | ClusterHeartbeatSender           |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node [akka.tcp://actorsystem@192.168.0.158:2551] - Heartbeat to [akka.tcp://actorsystem@192.168.0.157:2551]

2017-05-23 20:59:34,265 | DEBUG | -dispatcher-2401 | ClusterHeartbeatSender           |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node [akka.tcp://actorsystem@192.168.0.158:2551] - Heartbeat to [akka.tcp://actorsystem@192.168.0.157:2551]

2017-05-23 20:59:35,266 | DEBUG | -dispatcher-2405 | ClusterHeartbeatSender           |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node [akka.tcp://actorsystem@192.168.0.158:2551] - Heartbeat to [akka.tcp://actorsystem@192.168.0.157:2551]

2017-05-23 20:59:36,270 | DEBUG | -dispatcher-2401 | ClusterHeartbeatSender           |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node [akka.tcp://actorsystem@192.168.0.158:2551] - Heartbeat to [akka.tcp://actorsystem@192.168.0.157:2551]

2017-05-23 20:59:37,265 | DEBUG | -dispatcher-2401 | ClusterHeartbeatSender           |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node [akka.tcp://actorsystem@192.168.0.158:2551] - Heartbeat to [akka.tcp://actorsystem@192.168.0.157:2551]

2017-05-23 20:59:38,266 | DEBUG | -dispatcher-2408 | ClusterHeartbeatSender           |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node [akka.tcp://actorsystem@192.168.0.158:2551] - Heartbeat to [akka.tcp://actorsystem@192.168.0.157:2551]

2017-05-23 20:59:39,265 | DEBUG | -dispatcher-2408 | ClusterHeartbeatSender           |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node [akka.tcp://actorsystem@192.168.0.158:2551] - Heartbeat to [akka.tcp://actorsystem@192.168.0.157:2551]

2017-05-23 20:59:40,265 | DEBUG | -dispatcher-2405 | ClusterHeartbeatSender           |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node [akka.tcp://actorsystem@192.168.0.158:2551] - Heartbeat to [akka.tcp://actorsystem@192.168.0.157:2551]

2017-05-23 20:59:41,265 | DEBUG | -dispatcher-2405 | ClusterHeartbeatSender           |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node [akka.tcp://actorsystem@192.168.0.158:2551] - Heartbeat to [akka.tcp://actorsystem@192.168.0.157:2551]

2017-05-23 20:59:42,265 | DEBUG | -dispatcher-2408 | ClusterHeartbeatSender           |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node [akka.tcp://actorsystem@192.168.0.158:2551] - Heartbeat to [akka.tcp://actorsystem@192.168.0.157:2551]

2017-05-23 20:59:43,265 | DEBUG | -dispatcher-2401 | ClusterHeartbeatSender           |apply$mcV$sp() 77 | com.typesafe.akka.slf4j | Cluster Node [akka.tcp://actorsystem@192.168.0.158:2551] - Heartbeat to [akka.tcp://actorsystem@192.168.0.157:2551]

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.

To unsubscribe from this group and stop receiving emails from it, send an email to akka...@googlegroups.com.