hazelcast HealthMonitor entries "unknown gc count" and "unknown gc time" from the log shows increment in number.

[mudayku...@gmail.com]

Hello Forum,

Requesting your expert advice or recommendations on the below observation.

Problem: 

In the Webservices application server logs, there were transaction "Timed-Out" issue started on 09th June 2020 at 9:27:01, a total of 94 transactions were timed-out. There were also 74 entries with transactions "Roll-back" started at 9:28:33 AEST. These 74 roll-back transactions were from the 94 transactions which were timed-out. During the investigation, we have seen hazelcast HealthMonitor entries from the log are used to find any patterns or abnormalities.  The log duration is considered 03rd June 20 5:01 to 09th June 20 11:13. (Note: The application server was restarted on 6/3/20 5:01 AEST and the time-out issue started occurring on 6/9/20 09:27. So, the application was restarted again on 09th June 2020 at 11:14 ) The below-attached table contains the entries of hazelcast Health Monitor. 

The suspecting issues here are:
- The increase in heap memory usage and occupied at the max 98.13% 
- The values "unknown GC count" and "unknown GC time" parameters are unknown. The values of these parameters kept increasing between the two restart instances. 

Date

Time

physical memory free

heap memory used

heap memory free

heap memory used/total

heap memory used/max

unknown gc count

unknown gc time

load process

load system

load system Average

thread count

thread peakCount

Jun-03

5:01:50 AM

845.3M

1.7G

853.1M

67.42%

57.47%

326

4775ms

15.00%

73.00%

460.00%

125

127

Jun-09

11:13:29 AM

720.2M

2.9G

57.3M

98.13%

98.13%

169530

5029445ms

13.00%

65.00%

630.00%

127

135

Could you please advice what are the "unknown GC count" and "unknown GC time" here and why are they keep increasing between two restart instances of the application? Also, any recommendations to update/correct the configurations to resolve this problem?

[Ozan Kılıç]

Hi,

On some JVMs and environments, HealthMonitor cannot get the metrics for major and minor GCs so it logs them as unknown.gc. 

These metrics show that you have run out the heap space so JVM started heavy GC activities. 

You need to increase the heap size to get rid of this problem. 

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/1237b437-b403-4e2d-8177-1872f50b274fo%40googlegroups.com.

--

Ozan KILIC Solutions Architect - Support       hazelcast®

This message contains confidential information and is intended only for the individuals named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. If verification is required, please request a hard-copy version. -Hazelcast

[Uda!]

Thank you Ozan 

To unsubscribe from this group and stop receiving emails from it, send an email to haze...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/1237b437-b403-4e2d-8177-1872f50b274fo%40googlegroups.com.