plugin zookeeper, failed to invoke custom onModule method

[Joe Wardell]

Hi,

I've installed the zookeeper plugin for ES but on startup elasticsearch.log outputs the following warning, caused by an exception in the zookeeper plugin:

[2014-10-29 10:36:18,700][INFO ][node                     ] [Amelia Voght] version[1.3.4], pid[8782], build[a70f3cc/2014-09-30T09:07:17Z]

[2014-10-29 10:36:18,701][INFO ][node                     ] [Amelia Voght] initializing ...

[2014-10-29 10:36:18,724][INFO ][plugins                  ] [Amelia Voght] loaded [zookeeper], sites [head]

[2014-10-29 10:36:18,768][INFO ][org.apache.zookeeper.ZooKeeper] Client environment:zookeeper.version=3.3.6-1366786, built on 07/29/2012 06:22 GMT

[2014-10-29 10:36:18,768][INFO ][org.apache.zookeeper.ZooKeeper] Client environment:host.name=joewardellbox

[2014-10-29 10:36:18,768][INFO ][org.apache.zookeeper.ZooKeeper] Client environment:java.version=1.7.0_65

[2014-10-29 10:36:18,768][INFO ][org.apache.zookeeper.ZooKeeper] Client environment:java.vendor=Oracle Corporation

[2014-10-29 10:36:18,768][INFO ][org.apache.zookeeper.ZooKeeper] Client environment:java.home=/usr/lib/jvm/java-7-openjdk-amd64/jre

[2014-10-29 10:36:18,768][INFO ][org.apache.zookeeper.ZooKeeper] Client environment:java.class.path=:/usr/share/elasticsearch/lib/elasticsearch-1.3.4.jar:/usr/share/elasticsearch/lib/lucene-grouping-4.9.1.jar:/usr/share/elasticsearch/lib/log4j-1.2.17.jar:/usr/share/elasticsearch/lib/jna-4.1.0.jar:/usr/share/elasticsearch/lib/elasticsearch-1.3.4.jar:/usr/share/elasticsearch/lib/lucene-expressions-4.9.1.jar:/usr/share/elasticsearch/lib/lucene-suggest-4.9.1.jar:/usr/share/elasticsearch/lib/lucene-spatial-4.9.1.jar:/usr/share/elasticsearch/lib/lucene-memory-4.9.1.jar:/usr/share/elasticsearch/lib/groovy-all-2.3.2.jar:/usr/share/elasticsearch/lib/lucene-misc-4.9.1.jar:/usr/share/elasticsearch/lib/lucene-highlighter-4.9.1.jar:/usr/share/elasticsearch/lib/jts-1.13.jar:/usr/share/elasticsearch/lib/lucene-core-4.9.1.jar:/usr/share/elasticsearch/lib/lucene-queryparser-4.9.1.jar:/usr/share/elasticsearch/lib/spatial4j-0.4.1.jar:/usr/share/elasticsearch/lib/lucene-analyzers-common-4.9.1.jar:/usr/share/elasticsearch/lib/lucene-sandbox-4.9.1.jar:/usr/share/elasticsearch/lib/lucene-codecs-4.9.1.jar:/usr/share/elasticsearch/lib/lucene-join-4.9.1.jar:/usr/share/elasticsearch/lib/lucene-queries-4.9.1.jar:/usr/share/elasticsearch/lib/sigar/sigar-1.6.4.jar

[2014-10-29 10:36:18,769][INFO ][org.apache.zookeeper.ZooKeeper] Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib

[2014-10-29 10:36:18,769][INFO ][org.apache.zookeeper.ZooKeeper] Client environment:java.io.tmpdir=/tmp

[2014-10-29 10:36:18,769][INFO ][org.apache.zookeeper.ZooKeeper] Client environment:java.compiler=<NA>

[2014-10-29 10:36:18,769][INFO ][org.apache.zookeeper.ZooKeeper] Client environment:os.name=Linux

[2014-10-29 10:36:18,769][INFO ][org.apache.zookeeper.ZooKeeper] Client environment:os.arch=amd64

[2014-10-29 10:36:18,769][INFO ][org.apache.zookeeper.ZooKeeper] Client environment:os.version=3.13.0-36-generic

[2014-10-29 10:36:18,769][INFO ][org.apache.zookeeper.ZooKeeper] Client environment:user.name=elasticsearch

[2014-10-29 10:36:18,769][INFO ][org.apache.zookeeper.ZooKeeper] Client environment:user.home=/usr/share/elasticsearch

[2014-10-29 10:36:18,769][INFO ][org.apache.zookeeper.ZooKeeper] Client environment:user.dir=/

[2014-10-29 10:36:18,771][INFO ][org.apache.zookeeper.ZooKeeper] Initiating client connection, connectString=jazzy:2181,doom:2181,thrash:2181 sessionTimeout=60000 watcher=com.sonian.elasticsearch.zookeeper.client.ZooKeeperClientService$1@4c0147e3

[2014-10-29 10:36:18,810][INFO ][org.apache.zookeeper.ClientCnxn] Opening socket connection to server thrash/192.168.17.167:2181

[2014-10-29 10:36:18,818][INFO ][org.apache.zookeeper.ClientCnxn] Socket connection established to thrash/192.168.17.167:2181, initiating session

[2014-10-29 10:36:18,827][INFO ][org.apache.zookeeper.ClientCnxn] Session establishment complete on server thrash/192.168.17.167:2181, sessionid = 0x24954cfe0e1001c, negotiated timeout = 40000

[2014-10-29 10:36:18,842][INFO ][org.apache.zookeeper.ZooKeeper] Session: 0x24954cfe0e1001c closed

[2014-10-29 10:36:18,842][INFO ][org.apache.zookeeper.ClientCnxn] EventThread shut down

[2014-10-29 10:36:18,996][WARN ][plugins                  ] [Amelia Voght] plugin zookeeper, failed to invoke custom onModule method

java.lang.reflect.InvocationTargetException

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at org.elasticsearch.plugins.PluginsService.processModule(PluginsService.java:199)

at org.elasticsearch.plugins.PluginsModule.processModule(PluginsModule.java:61)

at org.elasticsearch.common.inject.Modules.processModules(Modules.java:64)

at org.elasticsearch.common.inject.ModulesBuilder.createInjector(ModulesBuilder.java:58)

at org.elasticsearch.node.internal.InternalNode.<init>(InternalNode.java:192)

at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:159)

at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:70)

at org.elasticsearch.bootstrap.Bootstrap.main(Bootstrap.java:203)

at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:32)

Caused by: java.lang.VerifyError: class com.sonian.elasticsearch.rest.zookeeper.RestZooKeeperStatusAction overrides final method handleRequest.(Lorg/elasticsearch/rest/RestRequest;Lorg/elasticsearch/rest/RestChannel;)V

at java.lang.ClassLoader.defineClass1(Native Method)

at java.lang.ClassLoader.defineClass(ClassLoader.java:800)

at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)

at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)

at java.net.URLClassLoader.access$100(URLClassLoader.java:71)

at java.net.URLClassLoader$1.run(URLClassLoader.java:361)

at java.net.URLClassLoader$1.run(URLClassLoader.java:355)

at java.security.AccessController.doPrivileged(Native Method)

at java.net.URLClassLoader.findClass(URLClassLoader.java:354)

at java.lang.ClassLoader.loadClass(ClassLoader.java:425)

at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)

at java.lang.ClassLoader.loadClass(ClassLoader.java:358)

at com.sonian.elasticsearch.plugin.zookeeper.ZooKeeperPlugin.onModule(ZooKeeperPlugin.java:54)

... 13 more

[2014-10-29 10:36:22,466][INFO ][node                     ] [Amelia Voght] initialized

[2014-10-29 10:36:22,467][INFO ][node                     ] [Amelia Voght] starting ...

[2014-10-29 10:36:22,682][INFO ][transport                ] [Amelia Voght] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/192.168.19.200:9300]}

[2014-10-29 10:36:22,705][INFO ][org.apache.zookeeper.ZooKeeper] Initiating client connection, connectString=jazzy:2181,doom:2181,thrash:2181 sessionTimeout=60000 watcher=com.sonian.elasticsearch.zookeeper.client.ZooKeeperClientService$1@1bc60ccc

[2014-10-29 10:36:22,706][INFO ][org.apache.zookeeper.ClientCnxn] Opening socket connection to server doom/192.168.17.151:2181

[2014-10-29 10:36:22,708][INFO ][org.apache.zookeeper.ClientCnxn] Socket connection established to doom/192.168.17.151:2181, initiating session

[2014-10-29 10:36:22,712][INFO ][org.apache.zookeeper.ClientCnxn] Session establishment complete on server doom/192.168.17.151:2181, sessionid = 0x34954cfef70001e, negotiated timeout = 40000

[2014-10-29 10:36:22,720][INFO ][discovery                ] [Amelia Voght] elasticsearch/Hu_T8A5yQJ2LbNyIlUwVEQ

After which ES never seems to have an elected master (http://localhost:9200/_status):

{

• error: "ClusterBlockException[blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];[SERVICE_UNAVAILABLE/2/no master];]",
• status: 503

}

Any ideas what would cause the exception?

Thanks for your help.