I'm using jmx exporter to expose Tomcat metrics, but got those errors,

[tangc...@gmail.com]

Could someone help me out please?

here is my config:

1. I'm running redhat 7

2. in setenv.sh, i have added 2 lines:
   Prometheus_JMX_OPTS="-javaagent:/opt/prometheus/lib/jmx_prometheus_javaagent-0.10.jar=7777:/opt/prometheus/conf/config.yml"
   JMX_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=3000 -Dcom.sun.management.jmxremote.rmi.port=3000 -Dcom.sun.management.jm
   xremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false"

3. modified JAVA_OPTS to add above 2 opts:
   JAVA_OPTS="-Xms${JVM_MINIMUM_MEMORY} -Xmx${JVM_MAXIMUM_MEMORY} ${JAVA_OPTS} ${OPC_JVM_ARGS} ${JVM_REQUIRED_ARGS} ${DISABLE_NOTIFICATIONS} ${JVM_SUPPOR
   T_RECOMMENDED_ARGS} ${JVM_EXTRA_ARGS} ${JIRA_HOME_MINUSD} ${JMX_OPTS} ${Prometheus_JMX_OPTS}"

4. downloaded javaagent.jar and placed in the folder

5. created config.yml file and placed in the folder. here is the file content:

---

hostPort: localhost:5555
username:
password:

rules:

• pattern: ".*"

6. restarted app,

7. queried the metrics http://localhost:7777/metrics
   jvm_memory_bytes_used{area="heap",} 1.349973384E9
   jvm_memory_bytes_used{area="nonheap",} 2.96595936E8
   jvm_memory_bytes_committed{area="heap",} 8.392278016E9
   jvm_memory_bytes_committed{area="nonheap",} 3.19315968E8
   jvm_memory_bytes_max{area="heap",} 8.392278016E9
   jvm_memory_bytes_max{area="nonheap",} -1.0
   jvm_memory_pool_bytes_used{pool="Code Cache",} 6.3392448E7
   jvm_memory_pool_bytes_used{pool="Metaspace",} 2.056872E8
   jvm_memory_pool_bytes_used{pool="Compressed Class Space",} 2.7516288E7
   jvm_memory_pool_bytes_used{pool="PS Eden Space",} 1.005291552E9
   jvm_memory_pool_bytes_used{pool="PS Survivor Space",} 1.3884524E8
   jvm_memory_pool_bytes_used{pool="PS Old Gen",} 2.05836592E8
   jvm_memory_pool_bytes_committed{pool="Code Cache",} 6.4159744E7
   jvm_memory_pool_bytes_committed{pool="Metaspace",} 2.22732288E8
   jvm_memory_pool_bytes_committed{pool="Compressed Class Space",} 3.2423936E7
   jvm_memory_pool_bytes_committed{pool="PS Eden Space",} 2.48774656E9
   jvm_memory_pool_bytes_committed{pool="PS Survivor Space",} 1.77733632E8
   jvm_memory_pool_bytes_committed{pool="PS Old Gen",} 5.726797824E9
   jvm_memory_pool_bytes_max{pool="Code Cache",} 2.5165824E8
   jvm_memory_pool_bytes_max{pool="Metaspace",} -1.0
   jvm_memory_pool_bytes_max{pool="Compressed Class Space",} 1.073741824E9
   jvm_memory_pool_bytes_max{pool="PS Eden Space",} 2.497708032E9
   jvm_memory_pool_bytes_max{pool="PS Survivor Space",} 1.77733632E8
   jvm_memory_pool_bytes_max{pool="PS Old Gen",} 5.726797824E9
   jmx_scrape_duration_seconds 0.001350404
   jmx_scrape_error 1.0
   jmx_config_reload_success_total 0.0
   jmx_config_reload_failure_total 0.0
   process_cpu_seconds_total 247.45
   process_start_time_seconds 1.504714538615E9
   process_open_fds 722.0
   process_max_fds 4096.0
   process_virtual_memory_bytes 2.0026785792E10
   process_resident_memory_bytes 3.706384384E9
   jvm_info{version="1.8.0_51-b16",vendor="Oracle Corporation",} 1.0
   jvm_threads_current 120.0
   jvm_threads_daemon 105.0
   jvm_threads_peak 121.0
   jvm_threads_started_total 160.0
   jvm_threads_deadlocked 0.0
   jvm_threads_deadlocked_monitor 0.0
   jvm_gc_collection_seconds_count{gc="PS Scavenge",} 12.0
   jvm_gc_collection_seconds_sum{gc="PS Scavenge",} 0.969
   jvm_gc_collection_seconds_count{gc="PS MarkSweep",} 5.0
   jvm_gc_collection_seconds_sum{gc="PS MarkSweep",} 1.431
   jvm_classes_loaded 40750.0
   jvm_classes_loaded_total 40756.0
   jvm_classes_unloaded_total 6.0

8. But from log file, I saw errors:
   06-Sep-2017 16:43:30.918 SEVERE [pool-1-thread-2] io.prometheus.jmx.shaded.io.prometheus.jmx.JmxCollector.collect JMX scrape failed: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.NoInitialContextException: Cannot instantiate class: org.apache.naming.java.javaURLContextFactory [Root exception is java.lang.ClassNotFoundException: org.apache.naming.java.javaURLContextFactory]
   at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:369)
   at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
   at io.prometheus.jmx.shaded.io.prometheus.jmx.JmxScraper.doScrape(JmxScraper.java:106)
   at io.prometheus.jmx.shaded.io.prometheus.jmx.JmxCollector.collect(JmxCollector.java:415)
   at io.prometheus.jmx.shaded.io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.findNextElement(CollectorRegistry.java:180)
   at io.prometheus.jmx.shaded.io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:213)
   at io.prometheus.jmx.shaded.io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:134)
   at io.prometheus.jmx.shaded.io.prometheus.client.exporter.common.TextFormat.write004(TextFormat.java:22)
   at io.prometheus.jmx.shaded.io.prometheus.client.exporter.HTTPServer$HTTPMetricHandler.handle(HTTPServer.java:43)
   at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:79)
   at sun.net.httpserver.AuthFilter.doFilter(AuthFilter.java:83)
   at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:82)
   at sun.net.httpserver.ServerImpl$Exchange$LinkHandler.handle(ServerImpl.java:675)
   at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:79)
   at sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:647)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   at java.lang.Thread.run(Thread.java:745)
   Caused by: javax.naming.NoInitialContextException: Cannot instantiate class: org.apache.naming.java.javaURLContextFactory [Root exception is java.lang.ClassNotFoundException: org.apache.naming.java.javaURLContextFactory]
   at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:674)
   at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:313)
   at javax.naming.InitialContext.init(InitialContext.java:244)
   at javax.naming.InitialContext.(InitialContext.java:216)
   at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1955)
   at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1924)
   at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:287)
   ... 17 more
   Caused by: java.lang.ClassNotFoundException: org.apache.naming.java.javaURLContextFactory
   at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
   at java.lang.Class.forName0(Native Method)
   at java.lang.Class.forName(Class.java:348)
   at com.sun.naming.internal.VersionHelper12.loadClass(VersionHelper12.java:72)
   at com.sun.naming.internal.VersionHelper12.loadClass(VersionHelper12.java:61)
   at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:672)
   ... 23 more

I searched around, but couldn't find any useful solution to fix the issue.

any assistance will be greatly appreciated!

[Meier]

Is there any context running in you tomcat?

[tangc...@gmail.com]

Hi Meier, Could you please elaborate further? Not get you mean "context". I'm monitoring JIRA  which is JAVA app. Sorry not sure how to answer you.

[Meier]

Tomcat context are well documented https://tomcat.apache.org/tomcat-7.0-doc/config/context.html...

I was simply guessing the error could mean that you use https://github.com/prometheus/jmx_exporter/blob/master/example_configs/tomcat.yml on a tomcat wich does not have a webapp installed or not installed at a location the exporter will find it.

[tangc...@gmail.com]

Hi Meier,

Thanks for shining here. Thanks again.

I'm going through tomcat context.

besides, my config file isn't tomcat.yml. my config file is 

---

hostPort: localhost:5555
username:
password:

rules:

• pattern: ".*"

[rjs...@gmail.com]

Shameless plug .....

Depending on the metrics you want you could also use my tomcat specific exporter. See https://github.com/nlighten/tomcat_exporter/

[mru...@gmail.com]

Your errors indicate that it's trying to access JMX on 5555, but you're exposing it on port 3000 so it would never work.

Try removing the hostPort, username, and password values from your config.yml. Since you're running it as a javaagent, you don't need those as it will read from the mBeans internally from the JVM.

--- snip ---

hostPort: localhost:5555

username:

password:

--- snip ---

[zhnn...@gmail.com]

Change
hostPort: localhost:5555

to
hostPort: localhost

It will work out

在 2017年9月7日星期四 UTC+8上午12:57:22，tangc...@gmail.com写道：

[sweetha...@gmail.com]

Could you pls let me know what resolved the error ? I am running as javaagent as well and getting the exact same error.

[zau...@gmail.com]

You have to remove hostPort, username and password fields.

[Jie Yang]

I delete hostPort ,username and password options in config.yml and still throw  this error.

envrionment:

jdk.1.8

tomcat 7.0.82

and jmx_exporter as a agent within tomcat start.

segment in catalina.sh

```

export CATALINA_OPTS="-javaagent:$PWD/jmx_exporter/jmx_prometheus_javaagent-0.3.2-SNAPSHOT.jar=5556:$PWD/jmx_exporter/tomcat.yml

                      -Djava.library.path=/usr/local/apr/lib

                      -Dcom.sun.management.jmxremote

                      -Djava.rmi.server.hostname=node3

                      -Dcom.sun.management.jmxremote.port=9999 

                      -Dcom.sun.management.jmxremote.rmi.port=9999

                      -Dcom.sun.management.jmxremote.authenticate=false"

```

and tomcat.yml

```

---

startDelaySeconds: 0

jmxUrl: service:jmx:rmi:///jndi/rmi://node3:9999/jmxrmi

ssl: false

lowercaseOutputLabelNames: true

lowercaseOutputName: true

whitelistObjectNames: ["java.lang:type=OperatingSystem"]

blacklistObjectNames: []

rules:

  - pattern: ".*"

```

any one help me?

[Jie Yang]

jmx_promethus_javaagent-0.10 is the same .

[Jie Yang]

I see .jmx exporter with tomcat need no hostPort and jmxUrl  

[yongdao xuy]

We are using jmx exporter to get metrics for java application, we have removed port/usrname/password filed in config.yaml, but still get IO exception error like below.

anyone can give us somelight for this?  thanks!!!

we are using jmx_javaagent-0.12.0.

1. config.yaml

----------------
startDelaySeconds: 0
hostPort: localhost
ssl: false
lowercaseOutputName: false
lowercaseOutputLabelNames: false
rules:
  - pattern: ".*"
--------------------

2. JAVA_OPTS 

-Xms1096m -Xmx1096m -javaagent:/usr/src/jmx_javaagent-0.12.0.jar=7676:/usr/src/config.yaml -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/CSSBackGroundTask/logs/common -Dfile.encoding=UTF-8 -Dlogging.config=/usr/local/hitv/css-back-ground-task-logback.xml -Dspring.config.location=/usr/local/hitv/application.yml

3. still see connection refuse error like below.

     2019-07-31 08:40:44,103 ERROR [pool-1-thread-3] [io.prometheus.jmx.shaded.io.prometheus.jmx.JmxCollector] - JMX scrape failed: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavail

ableException [Root exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exception is:

java.net.ConnectException: Connection refused (Connection refused)]

at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:369)

at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)

at io.prometheus.jmx.shaded.io.prometheus.jmx.JmxScraper.doScrape(JmxScraper.java:94)

at io.prometheus.jmx.shaded.io.prometheus.jmx.JmxCollector.collect(JmxCollector.java:468)

at io.prometheus.jmx.shaded.io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.findNextElement(CollectorRegistry.java:183)

at io.prometheus.jmx.shaded.io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:216)

at io.prometheus.jmx.shaded.io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:137)

at io.prometheus.jmx.shaded.io.prometheus.client.exporter.common.TextFormat.write004(TextFormat.java:22)

at io.prometheus.jmx.shaded.io.prometheus.client.exporter.HTTPServer$HTTPMetricHandler.handle(HTTPServer.java:59)

at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:79)

at sun.net.httpserver.AuthFilter.doFilter(AuthFilter.java:83)

at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:82)

at sun.net.httpserver.ServerImpl$Exchange$LinkHandler.handle(ServerImpl.java:675)

at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:79)

at sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:647)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

Caused by: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exception is:

java.net.ConnectException: Connection refused (Connection refused)]

at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:136)

at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:205)

at javax.naming.InitialContext.lookup(InitialContext.java:417)

at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1955)

at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1922)

at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:287)

... 17 more

Caused by: java.rmi.ConnectException: Connection refused to host: localhost; nested exception is:

java.net.ConnectException: Connection refused (Connection refused)

at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619)

at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)

at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)

at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:338)

at sun.rmi.registry.RegistryImpl_Stub.lookup(RegistryImpl_Stub.java:112)

at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:132)

... 22 more

Caused by: java.net.ConnectException: Connection refused (Connection refused)

at java.net.PlainSocketImpl.socketConnect(Native Method)

at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)

at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)

at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)

at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)

at java.net.Socket.connect(Socket.java:589)

at java.net.Socket.connect(Socket.java:538)

at java.net.Socket.<init>(Socket.java:434)

at java.net.Socket.<init>(Socket.java:211)

at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40)