Issue with enabling JMX in kafka brokers

[sammu...@gmail.com]

Hi all, 

We've been working on enabling JMX in kafka brokers. To accomplish the task, I've made all the necessary changes in kafka-run-class.sh and kafka-server-start.sh. However, when I run the command- " sh kafka-run-class.sh kafka.tools.JmxTool --jmx-url service:jmx:rmi:///jndi/rmi://<server_name>:9111/jmxrmi", I see the following stack trace:

Exception in thread "main" java.rmi.UnmarshalException: error unmarshalling return; nested exception is:

java.io.WriteAbortedException: writing aborted; java.io.NotSerializableException: mx4j.tools.adaptor.http.XSLTProcessor

at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:191)

at com.sun.jmx.remote.internal.PRef.invoke(Unknown Source)

at javax.management.remote.rmi.RMIConnectionImpl_Stub.getAttributes(Unknown Source)

at javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.getAttributes(RMIConnector.java:923)

at kafka.tools.JmxTool$$anonfun$queryAttributes$1.apply(JmxTool.scala:124)

at kafka.tools.JmxTool$$anonfun$queryAttributes$1.apply(JmxTool.scala:122)

at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59)

at scala.collection.immutable.List.foreach(List.scala:76)

at kafka.tools.JmxTool$.queryAttributes(JmxTool.scala:122)

at kafka.tools.JmxTool$.main(JmxTool.scala:102)

at kafka.tools.JmxTool.main(JmxTool.scala)

Caused by: java.io.WriteAbortedException: writing aborted; java.io.NotSerializableException: mx4j.tools.adaptor.http.XSLTProcessor

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)

at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1996)

at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1920)

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1797)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1349)

at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)

at java.util.ArrayList.readObject(ArrayList.java:696)

at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:622)

at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1042)

at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1896)

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1797)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1349)

at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)

at sun.rmi.server.UnicastRef.unmarshalValue(UnicastRef.java:324)

at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:173)

... 10 more

Caused by: java.io.NotSerializableException: mx4j.tools.adaptor.http.XSLTProcessor

at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1183)

at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)

at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)

at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)

at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)

at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)

at java.util.ArrayList.writeObject(ArrayList.java:673)

at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:622)

at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1012)

at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1495)

at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)

at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)

at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)

at sun.rmi.server.UnicastRef.marshalValue(UnicastRef.java:292)

at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:332)

at sun.rmi.transport.Transport$2.run(Transport.java:202)

at java.security.AccessController.doPrivileged(Native Method)

at sun.rmi.transport.Transport.serviceCall(Transport.java:198)

at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:564)

at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:825)

at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.access$400(TCPTransport.java:616)

at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$1.run(TCPTransport.java:681)

at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$1.run(TCPTransport.java:678)

at java.security.AccessController.doPrivileged(Native Method)

at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:678)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:701)

The kafka version is : 0.8.1. We are absolutely clueless as to why this has been happening. Appreciate your help!

Thank you!

[gerard...@dizzit.com]

Haven't use the tool myself, but it defaults to quering all objects, and some is giving an axception. You could try add --object-name  kafka.server.BrokerTopicMetrics   to see if this is indeed the problem.

[sammu...@gmail.com]

The stack trace below is what I see after I add --object-name kafka.server.BrokerTopicMetrics.

sh kafka-run-class.sh kafka.tools.JmxTool --jmx-url service:jmx:rmi:///jndi/rmi://<server_IP>:9111/jmxrmi --object-name kafka.server.BrokerTopicMetrics

Exception in thread "main" javax.management.MalformedObjectNameException: Key properties cannot be empty

at javax.management.ObjectName.construct(ObjectName.java:482)

at javax.management.ObjectName.<init>(ObjectName.java:1415)

at kafka.tools.JmxTool$$anonfun$1.apply(JmxTool.scala:85)

at kafka.tools.JmxTool$$anonfun$1.apply(JmxTool.scala:85)

at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:233)

at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:233)

at scala.collection.Iterator$class.foreach(Iterator.scala:772)

at scala.collection.JavaConversions$JIteratorWrapper.foreach(JavaConversions.scala:573)

at scala.collection.IterableLike$class.foreach(IterableLike.scala:73)

at scala.collection.JavaConversions$JListWrapper.foreach(JavaConversions.scala:615)

at scala.collection.TraversableLike$class.map(TraversableLike.scala:233)

at scala.collection.JavaConversions$JListWrapper.map(JavaConversions.scala:615)

at kafka.tools.JmxTool$.main(JmxTool.scala:85)

at kafka.tools.JmxTool.main(JmxTool.scala)

[gerard...@dizzit.com]

Sorry don't know how this tool should work.