Issue with JMX on a newly created node

[Thales Ceolin]

I've been using cassandra-reaper successfully until I decommissioned a single node (out of 5) and joined with a new one.

Cassandra Notetool says that everythings runs fine. 

JMX is enabled at the new node.

I can nc/telnet the host/port of the JMX port on the new node from where cassandra-reaper is running.

I'm getting the following stacktrace:

ERROR  [pool-12-thread-4] i.c.j.JmxConnectionFactory - Failed creating a new JMX connection to 10.10.3.21:7199 

java.lang.RuntimeException: io.cassandrareaper.ReaperException: Failure when establishing JMX connection to 10.10.3.21:7199

        at io.cassandrareaper.jmx.JmxConnectionFactory$JmxConnectionProvider.apply(JmxConnectionFactory.java:222)

        at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660)

        at io.cassandrareaper.jmx.JmxConnectionFactory.connectImpl(JmxConnectionFactory.java:102)

        at io.cassandrareaper.jmx.JmxConnectionFactory.connectAny(JmxConnectionFactory.java:137)

        at io.cassandrareaper.jmx.ClusterFacade.connectImpl(ClusterFacade.java:802)

        at io.cassandrareaper.jmx.ClusterFacade.connect(ClusterFacade.java:786)

        at io.cassandrareaper.service.SegmentRunner.lambda$getNodeMetrics$0(SegmentRunner.java:557)

        at java.util.concurrent.FutureTask.run(FutureTask.java:266)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

Caused by: io.cassandrareaper.ReaperException: Failure when establishing JMX connection to 10.10.3.21:7199

        at io.cassandrareaper.jmx.JmxProxyImpl.connect(JmxProxyImpl.java:264)

        at io.cassandrareaper.jmx.JmxProxyImpl.connect(JmxProxyImpl.java:168)

        at io.cassandrareaper.jmx.JmxConnectionFactory$JmxConnectionProvider.apply(JmxConnectionFactory.java:213)

        ... 10 common frames omitted

Caused by: java.util.concurrent.ExecutionException: java.rmi.NoSuchObjectException: no such object in table

        at java.util.concurrent.FutureTask.report(FutureTask.java:122)

        at java.util.concurrent.FutureTask.get(FutureTask.java:206)

        at io.cassandrareaper.jmx.JmxProxyImpl.connectWithTimeout(JmxProxyImpl.java:282)

        at io.cassandrareaper.jmx.JmxProxyImpl.connect(JmxProxyImpl.java:221)

        ... 12 common frames omitted

Caused by: java.rmi.NoSuchObjectException: no such object in table

        at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:283)

        at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:260)

        at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:161)

        at javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown Source)

        at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2430)

        at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:308)

        at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)

        at io.cassandrareaper.jmx.JmxProxyImpl.lambda$connectWithTimeout$0(JmxProxyImpl.java:281)

        ... 4 common frames omitted

INFO   [pool-12-thread-4] i.c.j.JmxConnectionFactory - Unreachable host: Failure when establishing JMX connection to 10.10.3.21:7199: java.rmi.NoSuchObjectException: no such object in table 

I've googled around and could find someone mentioning the /etc/hosts but my hosts file looks OK.

Any thoughts or things to look into?

Many thanks.

[Reid Pinchback]

At a guess I would wonder if the new node is blocking the RMI port, like if the new node has different firewall/iptables rules not exposing ephemeral ports and you haven’t pinned the RMI port to one that is explicitly exposed.

 

Note that the above observation I’ve seen to be true when using JMX over SSL, I don’t recall if it applies if you aren’t using SSL.

 

From: <tlp-apache-cassa...@googlegroups.com> on behalf of Thales Ceolin <tha...@comet.ml>
Date: Thursday, December 12, 2019 at 2:19 PM
To: TLP Apache Cassandra Reaper users <tlp-apache-cassa...@googlegroups.com>
Subject: Issue with JMX on a newly created node

 

Message from External Sender

--
You received this message because you are subscribed to the Google Groups "TLP Apache Cassandra Reaper users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-r...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tlp-apache-cassandra-reaper-users/e68c7861-8df0-4d33-9565-1581db9dec4b%40googlegroups.com.