JMX monitoring from local datadog agent can't get access

[Pluto]

I am trying to get the datadog agent to pull JMX info from each Payara instance. I have enabled JMX Monitoring logging in the DAS. In the server.log I see that it is started

JMXStartupService has started JMXConnector on JMXService URL service:jmx:rmi://0.0.0.0:28686/jndi/rmi://0.0.0.0:28686/jmxrmi]]

I used the url lilsted to enter into the datadog agent

jmx_url: "service:jmx:rmi://0.0.0.0:28686/jndi/rmi://0.0.0.0:28686/jmxrmi"

When it tries to connect it gets this error, do I need to allow "remote" access to pull this information?

      [ERROR]: 'Cannot connect to instance service:jmx:rmi://0.0.0.0:28686/jndi/rmi://0.0.0.0:28686/jmxrmi. java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: non-JRMP server at remote endpoint]' collected 0 metrics

      - Collected 0 metrics, 0 events & 0 service checks

[Pluto]

Answering my own question in case it helps anyone. You need to add these to the JVM Settings -> JVM Options in the config that your instance is using.

1 2 3 4 5

-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=<jmx_remote_port, in my case I used 38686> -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false

Then the datadog jmx.yaml file should look like this

--------------------------------------------------------------

init_config:

instances:

  - host: localhost

    jmx_url: "service:jmx:rmi://0.0.0.0:38686/jndi/rmi://0.0.0.0:38686/jmxrmi"

    tools_jar_path: /opt/jdk/jdk1.8.0_151/lib/tools.jar

    refresh_beans: 600 # optional (in seconds)

    conf:

      - include:

          domain: java.lang

          bean:

            - java.lang:type=Memory,name=Memory

          attribute:

            HeapMemoryUsage:

              metric_type: gauge

              alias: jmx.memory.heap_usage

---------------------------------------------------------------------------------------------

[ondrej....@gmail.com]

This is certainly a solution that works.

 

Just note that it's not secure because it opens JMX for anybody and JMX can also be used to manage the server and change it's configuration. Therefore use this solution only in a private network.

 

Ondrej

--
You received this message because you are subscribed to the Google Groups "Payara Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to payara-forum...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/payara-forum/8f811b28-50ad-4f11-8439-8c556f68d7bd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

 

[Ondro Mihályi]

The proper way to connect to a remote JMX interface of Payara Server is to use SSL with a proper certificate accepted by the server. This is for security reasons because JMX doesn't provide only monitoring data but also enables to manage the server remotely and opening the JMX port without SSL is insecure.

Pluto's solution works because it instructs JVM to open a new JMX interface over an unsecured connection. The new JMX interface provides exactly the same thing as the Payra Server interface but it's not secured. Therefore use this solution only in a private network.

Here's the instructions how to access the secured JMX interface properly: https://groups.google.com/d/msg/payara-forum/Jg_or-taPAk/meLyHWgnBAAJ
It requires copying Payara server's certificate to the client's keystore and truststore and not all clients support access to JMX with SSL. I've had problems configuring Zabbix to connect using SSL in the past and the same may apply to Datadog. Therefore it's possible that with Datadog there's no better solution than what Pluto described.

Ondro

To unsubscribe from this group and stop receiving emails from it, send an email to payara-forum+unsubscribe@googlegroups.com.