Problem using the "Separate" execution strategy

[Yann Cézard]

Hi,

I am using jmxtrans to collect JMX mectrics from several docker containers in a cluster, using some confd glue magic

(confd gets a list of containers that jmxtrans has to monitor and generate the config file).

The data is sent to an influxdb database.

I am using this for more then a year, but sometimes I have problems with some containers which stopped being monitored.

I even have one deployment where jmxtrans just collect 5 times (every 30 secondes) and just stop doing anything, while stil running,

and nothing helpful in the logs (at TRACE level).

So I ended up upgrading jmxtrans to the latest release (271) as it fixes several problem like this, but with no success (still the same problem).

After reading through the doc, I found about the executor strategies (https://github.com/jmxtrans/jmxtrans/wiki/Configuration), but

I have not much success with it, it is just worst as jmxtrans just do nothing at all and logs a lot of error.

The command line is :

/usr/local/openjdk-8/bin/java -server -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Djmxtrans.log.level=TRACE -Djmxtrans.log.dir=/var/log/jmxtrans -Xms256m -Xmx256m -XX:PermSize=384m -XX:MaxPermSize=384m -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=2101 -jar /usr/share/jmxtrans/lib/jmxtrans-all.jar -e -f conf/config.yml -s 30 -c false --use-separate-executors

The error is :

2020-07-29 15:15:15 [jmxtrans-scheduler-0] DEBUG c.g.j.scheduler.ServerScheduler - +++++ Started server job Server(pid=null, host=10.42.64.58, port=9010, url=service:jmx:rmi:///jndi/rmi://10.42.64.58:9010/jmxrmi, cronExpression=null, numQueryThreads=0)

2020-07-29 15:15:15 [jmxtrans-scheduler-0] WARN  c.g.j.scheduler.ServerScheduler - +++++ Failed server job Server(pid=null, host=10.42.64.58, port=9010, url=service:jmx:rmi:///jndi/rmi://10.42.64.58:9010/jmxrmi, cronExpression=null, numQueryThreads=0)

java.lang.NullPointerException: null

at com.googlecode.jmxtrans.scheduler.ServerCommand.run(ServerCommand.java:67)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

The full output is jmxtrans.out (joined to this message).

The config.yml is joined to (it is generated really naively, we ask for every possible measurement on every kind of JVM, even if they don't expose the Bean, that might be one of the original problem I guess...)

The question is now, why does use-seperator-executors just break everything ? Is it still supported ?