Which metrics should be used for GC majors (Jboss EAP) ?

[inter...@gmail.com]

Hi,

I use JMX Exporter v0.3.1 in Openshift with Prometheus and Grafana.
I collect many metrics. I use the "wildfly-10.yaml" configuration.

The jboss version is "JBoss EAP 6.4.19.GA".

Currently I have two graphs about GC :

• GC time / 1 min. rate for PS Scavenge and PS MarkSweep (regex in grafana : irate(jvm_gc_collection_seconds_sum{instance=~"$instance"}[1m]))
• GC count for PS Scavenge and PS MarkSweep (regex in grafana : jvm_gc_collection_seconds_count{instance=~"$instance"})

I'd like to have a graph with only major GC. But unfortunately I don't know which metrics I can use or how to modify the regexs above.

Currently I am collecting this data about JVM :

# HELP jmx_scrape_duration_seconds Time this JMX scrape took, in seconds.



# TYPE jmx_scrape_duration_seconds gauge

jmx_scrape_duration_seconds

# HELP jmx_scrape_error Non-zero if this scrape failed.

# TYPE jmx_scrape_error gauge

jmx_scrape_error

# HELP jvm_gc_collection_seconds Time spent in a given JVM garbage collector in seconds.

# TYPE jvm_gc_collection_seconds summary

jvm_gc_collection_seconds_count{gc="PS Scavenge",}

jvm_gc_collection_seconds_sum{gc="PS Scavenge",}

jvm_gc_collection_seconds_count{gc="PS MarkSweep",}

jvm_gc_collection_seconds_sum{gc="PS MarkSweep",}

# HELP jmx_config_reload_success_total Number of times configuration have successfully been reloaded.

# TYPE jmx_config_reload_success_total counter

jmx_config_reload_success_total

# HELP jmx_config_reload_failure_total Number of times configuration have failed to be reloaded.

# TYPE jmx_config_reload_failure_total counter

jmx_config_reload_failure_total

# HELP jvm_info JVM version info

# TYPE jvm_info gauge

jvm_info{version="1.8.0_161-b14",vendor="Oracle Corporation",runtime="OpenJDK Runtime Environment",}

# HELP jvm_classes_loaded The number of classes that are currently loaded in the JVM

# TYPE jvm_classes_loaded gauge

jvm_classes_loaded

# HELP jvm_classes_loaded_total The total number of classes that have been loaded since the JVM has started execution

# TYPE jvm_classes_loaded_total counter

jvm_classes_loaded_total

# HELP jvm_classes_unloaded_total The total number of classes that have been unloaded since the JVM has started execution

# TYPE jvm_classes_unloaded_total counter

jvm_classes_unloaded_total

# HELP jvm_buffer_pool_used_bytes Used bytes of a given JVM buffer pool.

# TYPE jvm_buffer_pool_used_bytes gauge

jvm_buffer_pool_used_bytes{pool="direct",}

jvm_buffer_pool_used_bytes{pool="mapped",}

# HELP jvm_buffer_pool_capacity_bytes Bytes capacity of a given JVM buffer pool.

# TYPE jvm_buffer_pool_capacity_bytes gauge

jvm_buffer_pool_capacity_bytes{pool="direct",}

jvm_buffer_pool_capacity_bytes{pool="mapped",}

# HELP jvm_buffer_pool_used_buffers Used buffers of a given JVM buffer pool.

# TYPE jvm_buffer_pool_used_buffers gauge

jvm_buffer_pool_used_buffers{pool="direct",}

jvm_buffer_pool_used_buffers{pool="mapped",}

# HELP jvm_threads_current Current thread count of a JVM

# TYPE jvm_threads_current gauge

jvm_threads_current

# HELP jvm_threads_daemon Daemon thread count of a JVM

# TYPE jvm_threads_daemon gauge

jvm_threads_daemon

# HELP jvm_threads_peak Peak thread count of a JVM

# TYPE jvm_threads_peak gauge

jvm_threads_peak

# HELP jvm_threads_started_total Started thread count of a JVM

# TYPE jvm_threads_started_total counter

jvm_threads_started_total

# HELP jvm_threads_deadlocked Cycles of JVM-threads that are in deadlock waiting to acquire object monitors or ownable synchronizers

# TYPE jvm_threads_deadlocked gauge

jvm_threads_deadlocked

# HELP jvm_threads_deadlocked_monitor Cycles of JVM-threads that are in deadlock waiting to acquire object monitors

# TYPE jvm_threads_deadlocked_monitor gauge

jvm_threads_deadlocked_monitor

# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.

# TYPE process_cpu_seconds_total counter

process_cpu_seconds_total

# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.

# TYPE process_start_time_seconds gauge

process_start_time_seconds

# HELP process_open_fds Number of open file descriptors.

# TYPE process_open_fds gauge

process_open_fds

# HELP process_max_fds Maximum number of open file descriptors.

# TYPE process_max_fds gauge

process_max_fds

# HELP process_virtual_memory_bytes Virtual memory size in bytes.

# TYPE process_virtual_memory_bytes gauge

process_virtual_memory_bytes

# HELP process_resident_memory_bytes Resident memory size in bytes.

# TYPE process_resident_memory_bytes gauge

process_resident_memory_bytes

# HELP jvm_memory_bytes_used Used bytes of a given JVM memory area.

# TYPE jvm_memory_bytes_used gauge

jvm_memory_bytes_used{area="heap",}

jvm_memory_bytes_used{area="nonheap",}

# HELP jvm_memory_bytes_committed Committed (bytes) of a given JVM memory area.

# TYPE jvm_memory_bytes_committed gauge

jvm_memory_bytes_committed{area="heap",}

jvm_memory_bytes_committed{area="nonheap",}

# HELP jvm_memory_bytes_max Max (bytes) of a given JVM memory area.

# TYPE jvm_memory_bytes_max gauge

jvm_memory_bytes_max{area="heap",}

jvm_memory_bytes_max{area="nonheap",}

# HELP jvm_memory_bytes_init Initial bytes of a given JVM memory area.

# TYPE jvm_memory_bytes_init gauge

jvm_memory_bytes_init{area="heap",}

jvm_memory_bytes_init{area="nonheap",}

# HELP jvm_memory_pool_bytes_used Used bytes of a given JVM memory pool.

# TYPE jvm_memory_pool_bytes_used gauge

jvm_memory_pool_bytes_used{pool="Code Cache",}

jvm_memory_pool_bytes_used{pool="Metaspace",}

jvm_memory_pool_bytes_used{pool="Compressed Class Space",}

jvm_memory_pool_bytes_used{pool="PS Eden Space",}

jvm_memory_pool_bytes_used{pool="PS Survivor Space",}

jvm_memory_pool_bytes_used{pool="PS Old Gen",}

# HELP jvm_memory_pool_bytes_committed Committed bytes of a given JVM memory pool.

# TYPE jvm_memory_pool_bytes_committed gauge

jvm_memory_pool_bytes_committed{pool="Code Cache",}

jvm_memory_pool_bytes_committed{pool="Metaspace",}

jvm_memory_pool_bytes_committed{pool="Compressed Class Space",}

jvm_memory_pool_bytes_committed{pool="PS Eden Space",}

jvm_memory_pool_bytes_committed{pool="PS Survivor Space",}

jvm_memory_pool_bytes_committed{pool="PS Old Gen",}

# HELP jvm_memory_pool_bytes_max Max bytes of a given JVM memory pool.

# TYPE jvm_memory_pool_bytes_max gauge

jvm_memory_pool_bytes_max{pool="Code Cache",}

jvm_memory_pool_bytes_max{pool="Metaspace",}

jvm_memory_pool_bytes_max{pool="Compressed Class Space",}

jvm_memory_pool_bytes_max{pool="PS Eden Space",}

jvm_memory_pool_bytes_max{pool="PS Survivor Space",}

jvm_memory_pool_bytes_max{pool="PS Old Gen",}

# HELP jvm_memory_pool_bytes_init Initial bytes of a given JVM memory pool.

# TYPE jvm_memory_pool_bytes_init gauge

jvm_memory_pool_bytes_init{pool="Code Cache",}

jvm_memory_pool_bytes_init{pool="Metaspace",}

jvm_memory_pool_bytes_init{pool="Compressed Class Space",}

jvm_memory_pool_bytes_init{pool="PS Eden Space",}

jvm_memory_pool_bytes_init{pool="PS Survivor Space",}

jvm_memory_pool_bytes_init{pool="PS Old Gen",}

Do you think I can graph major GCs with any of these metrics ?

If not, can you tell me what metrics I need to collect?

Thanks in advance for your help !

[Brian Brazil]

On 18 July 2018 at 15:14, <inter...@gmail.com> wrote:

Hi,

I use JMX Exporter v0.3.1 in Openshift with Prometheus and Grafana.
I collect many metrics. I use the "wildfly-10.yaml" configuration.

The jboss version is "JBoss EAP 6.4.19.GA".

Currently I have two graphs about GC :

• GC time / 1 min. rate for PS Scavenge and PS MarkSweep (regex in grafana : irate(jvm_gc_collection_seconds_sum{instance=~"$instance"}[1m]))
• GC count for PS Scavenge and PS MarkSweep (regex in grafana : jvm_gc_collection_seconds_count{instance=~"$instance"})

I'd like to have a graph with only major GC. But unfortunately I don't know which metrics I can use or how to modify the regexs above.

Currently I am collecting this data about JVM :

# HELP jmx_scrape_duration_seconds Time this JMX scrape took, in seconds.



# TYPE jmx_scrape_duration_seconds gauge

jmx_scrape_duration_seconds

# HELP jmx_scrape_error Non-zero if this scrape failed.

# TYPE jmx_scrape_error gauge

jmx_scrape_error

# HELP jvm_gc_collection_seconds Time spent in a given JVM garbage collector in seconds.

# TYPE jvm_gc_collection_seconds summary

jvm_gc_collection_seconds_count{gc="PS Scavenge",}

jvm_gc_collection_seconds_sum{gc="PS Scavenge",}

 

jvm_gc_collection_seconds_count{gc="PS MarkSweep",}

This varies depending on the JVM you're using and GC settings, but this is probably the one you want.

Brian

 

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/3f7fae2c-4d01-44a4-a1f6-6cfcbde39ef8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io

[inter...@gmail.com]

Hi Brian,

Thanks for your help !

Currently, we do not use -XX:+UseParallelOldGC.

This option is not enabled by default in the Jboss EAP Openshift image. I don't know if it's really interesting to activate it in Openshift ?

On our monolithic Jboss (in domain), the option is indeed activated.

Currently, the JVM options are as follows :

-server
-XX:+UseCompressedOops
-verbose:gc
-Xloggc:/opt/eap/standalone/log/gc.log
-XX:+PrintGCDetails
-XX:+PrintGCDateStamps
-XX:+UseGCLogFileRotation
-XX:NumberOfGCLogFiles=5
-XX:GCLogFileSize=3M
-XX:-TraceClassUnloading
-Xms1536m
-Xmx1536m
-Djava.net.preferIPv4Stack=true
-Djboss.modules.system.pkgs=org.jboss.logmanager,jdk.nashorn.api
-Djava.awt.headless=true
-Djboss.modules.policy-permissions=true
-XX:+UseParallelGC
-XX:MinHeapFreeRatio=20
-XX:MaxHeapFreeRatio=40
-XX:GCTimeRatio=4
-XX:AdaptiveSizePolicyWeight=90
-XX:MaxMetaspaceSize=256m
-XX:ParallelGCThreads=3
-Djava.util.concurrent.ForkJoinPool.common.parallelism=3
-XX:CICompilerCount=2
-XX:+ExitOnOutOfMemoryError
-Djava.security.egd=file:/dev/./urandom
-javaagent:/opt/prometheus/jmx_prom_javaagent.jar=9779:/opt/prometheus/prom-config.yml
-javaagent:/opt/jolokia/jolokia.jar=config=/opt/jolokia/etc/jolokia.properties -Xbootclasspath/p:/opt/eap/jboss-modules.jar:/opt/eap/modules/system/layers/base/.overlays/layer-base-jboss-eap-6.4.19.CP/org/jboss/logmanager/main/jboss-logmanager-1.5.8.Final-redhat-1.jar:/opt/eap/modules/system/layers/base/org/jboss/logmanager/ext/main/javax.json-1.0.4.jar:/opt/eap/modules/system/layers/base/org/jboss/logmanager/ext/main/jboss-logmanager-ext-1.0.0.Alpha2-redhat-1.jar -Djava.util.logging.manager=org.jboss.logmanager.LogManager
-Dorg.jboss.boot.log.file=/opt/eap/standalone/log/server.log
-Dlogging.configuration=file:/opt/eap/standalone/configuration/logging.properties
-jar /opt/eap/jboss-modules.jar
-mp /opt/eap/modules
-jaxpmodule javax.xml.jaxp-provider org.jboss.as.standalone
-Djboss.home.dir=/opt/eap
-Djboss.server.base.dir=/opt/eap/standalone
-c standalone-openshift.xml
-bmanagement 127.0.0.1
-b 10.241.27.107
-Djboss.node.name=xxxxxx

Thanks,

Best regards.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/3f7fae2c-4d01-44a4-a1f6-6cfcbde39ef8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io

[Brian Brazil]

On 19 July 2018 at 13:31, <inter...@gmail.com> wrote:

Hi Brian,

Thanks for your help !

Currently, we do not use -XX:+UseParallelOldGC.

This option is not enabled by default in the Jboss EAP Openshift image. I don't know if it's really interesting to activate it in Openshift ?

That sounds like a question for the maintainer of that image.

Brian

 

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/5fd76a6f-1757-4b41-8d32-a1d848fc84e4%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io