Heartbeat Manager observing clock jumps

[may.m...@gmail.com]

I'm seeing time gaps in my system logs followed by this message:

Aug 18, 2016 7:08:43 AM com.hazelcast.cluster.impl.ClusterHeartbeatManager

INFO: [10.13.12.107]:7019 [topic-group_Production] [3.6] System clock apparently jumped from 2016-08-18T07:07:55.983 to 2016-08-18T07:08:43.222 since last heartbeat (+42239 ms)

so my logs have a gap in them where nothing happens for 43 seconds followed by a hazelcast message saying the system clock has jumped 43 seconds. The system clocks haven't been changed out from under my process so I was wondering if anyone else has seen this behavior. Is it possible that the Hazelcast Heartbeats can halt or disrupt a process while it waits for a response or something? Or what other sorts of problems are co-related to this.

[Noctarius]

Hey,

Are you running on virtualized environments where you might have automatic snapshotting? Apart from that maybe GC?

Chris

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.
To post to this group, send email to haze...@googlegroups.com.
Visit this group at https://groups.google.com/group/hazelcast.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/af589570-6a12-4423-b011-eb6fad74e755%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Michael Pilone]

I've seen similar messages occasionally in a VM environment. We've had a number of "discussions" with our sys-admins about machines appearing to pause for short periods but no clear resolution. We also just recently turned on garbage collection logging and switched to the new G1 garbage collector to see if that makes a difference. So far we don't have anything conclusive but I would look at these two areas of your system first.

-mike

[may.m...@gmail.com]

Yeah thanks you guys, it does look like it was a Garbage Collection issue.