Intermittent error while using Jetty

[Parag Dere]

Hi all,

I am seeing a very weird behavior while using the ES-Jetty plugin.

My setup is on a windows box, and have set up elastic search as a windows service running under my credentials. Using the latest build of ES, and Jetty plugin.

When i start the service, ES works @ times (with the required authentication) and fails @ times.

I get HTTP 500 error whenever ES fails to run properly.

On checking the log file it complains about 

[WARN ][org.eclipse.jetty.server.AbstractHttpConnection] [Executioner] /

java.lang.NoClassDefFoundError: com/sonian/elasticsearch/http/jetty/JettyHttpServerRestRequest

at com.sonian.elasticsearch.http.jetty.handler.JettyHttpServerTransportHandler.handle(JettyHttpServerTransportHandler.java:66)

What i fail to understand is this behavior is intermittent. If the class doesnt exist i should get the error all the time, which is not the case.

Has anyone seen such a problem before ?

regards,

-Parag

[imotov]

Yes, we saw a few reports but I am unable to reproduce the issue. Could you post a complete stack? How many connections did you have open when this error occurred? Which windows service wrapper did you use? 

[Parag Dere]

Hi,

I restarted my elasticsearch service and it started normally.

The log file contents are as below:

[2012-08-01 07:15:56,398][WARN ][bootstrap                ] jvm uses the client vm, make sure to run `java` with the server vm for best performance by adding `-server` to the command line

[2012-08-01 07:15:56,403][INFO ][node                     ] [Warhawk] {0.19.8}[3124]: initializing ...

[2012-08-01 07:15:56,484][INFO ][plugins                  ] [Warhawk] loaded [jetty], sites [bigdesk, head]

[2012-08-01 07:15:59,555][INFO ][node                     ] [Warhawk] {0.19.8}[3124]: initialized

[2012-08-01 07:15:59,555][INFO ][node                     ] [Warhawk] {0.19.8}[3124]: starting ...

[2012-08-01 07:15:59,803][INFO ][transport                ] [Warhawk] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/192.168.2.2:9300]}

[2012-08-01 07:16:03,181][INFO ][cluster.service          ] [Warhawk] new_master [Warhawk][V8EIJFCMR-K4NXFSXGmwMw][inet[/192.168.2.2:9300]], reason: zen-disco-join (elected_as_master)

[2012-08-01 07:16:03,255][INFO ][discovery                ] [Warhawk] PaRaG/V8EIJFCMR-K4NXFSXGmwMw

[2012-08-01 07:16:03,677][INFO ][org.eclipse.jetty.server.Server] [Warhawk] jetty-8.1.4.v20120524

[2012-08-01 07:16:03,763][INFO ][org.eclipse.jetty.server.AbstractConnector] [Warhawk] Started SelectChann...@0.0.0.0:9200

[2012-08-01 07:16:03,906][INFO ][gateway                  ] [Warhawk] recovered [5] indices into cluster_state

[2012-08-01 07:16:04,027][INFO ][http                     ] [Warhawk] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/192.168.2.2:9200]}

[2012-08-01 07:16:04,027][INFO ][node                     ] [Warhawk] {0.19.8}[3124]: started

soon afterwards, i restarted the service and then i got the error again.

[2012-08-01 07:19:41,747][WARN ][bootstrap                ] jvm uses the client vm, make sure to run `java` with the server vm for best performance by adding `-server` to the command line

[2012-08-01 07:19:41,751][INFO ][node                     ] [Link] {0.19.8}[1196]: initializing ...

[2012-08-01 07:19:41,766][INFO ][plugins                  ] [Link] loaded [jetty], sites [bigdesk, head]

[2012-08-01 07:19:44,384][INFO ][node                     ] [Link] {0.19.8}[1196]: initialized

[2012-08-01 07:19:44,384][INFO ][node                     ] [Link] {0.19.8}[1196]: starting ...

[2012-08-01 07:19:44,643][INFO ][transport                ] [Link] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/192.168.2.2:9300]}

[2012-08-01 07:19:48,074][INFO ][cluster.service          ] [Link] new_master [Link][gCJHFIApQSmnxugHpLHe7g][inet[/192.168.2.2:9300]], reason: zen-disco-join (elected_as_master)

[2012-08-01 07:19:48,099][INFO ][discovery                ] [Link] PaRaG/gCJHFIApQSmnxugHpLHe7g

[2012-08-01 07:19:48,304][INFO ][org.eclipse.jetty.server.Server] [Link] jetty-8.1.4.v20120524

[2012-08-01 07:19:48,369][INFO ][org.eclipse.jetty.server.AbstractConnector] [Link] Started SelectChann...@0.0.0.0:9200

[2012-08-01 07:19:48,631][INFO ][http                     ] [Link] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/192.168.2.2:9200]}

[2012-08-01 07:19:48,631][INFO ][node                     ] [Link] {0.19.8}[1196]: started

[2012-08-01 07:19:48,756][INFO ][gateway                  ] [Link] recovered [5] indices into cluster_state

[2012-08-01 07:20:00,736][WARN ][org.eclipse.jetty.server.AbstractHttpConnection] [Link] /

java.lang.NoClassDefFoundError: com/sonian/elasticsearch/http/jetty/JettyHttpServerRestRequest

at com.sonian.elasticsearch.http.jetty.handler.JettyHttpServerTransportHandler.handle(JettyHttpServerTransportHandler.java:66)

at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:499)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111)

at org.eclipse.jetty.server.Server.handle(Server.java:350)

at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:454)

at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:890)

at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:944)

at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:630)

at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:230)

at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:77)

at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:620)

at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:46)

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:603)

at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:538)

at java.lang.Thread.run(Unknown Source)

Caused by: java.lang.ClassNotFoundException: com.sonian.elasticsearch.http.jetty.JettyHttpServerRestRequest

at java.net.URLClassLoader$1.run(Unknown Source)

at java.net.URLClassLoader$1.run(Unknown Source)

at java.security.AccessController.doPrivileged(Native Method)

at java.net.URLClassLoader.findClass(Unknown Source)

at java.lang.ClassLoader.loadClass(Unknown Source)

at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)

at java.lang.ClassLoader.loadClass(Unknown Source)

... 15 more

[2012-08-01 07:20:03,023][WARN ][org.eclipse.jetty.server.AbstractHttpConnection] [Link] /

java.lang.NoClassDefFoundError: com/sonian/elasticsearch/http/jetty/JettyHttpServerRestRequest

at com.sonian.elasticsearch.http.jetty.handler.JettyHttpServerTransportHandler.handle(JettyHttpServerTransportHandler.java:66)

at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:499)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111)

at org.eclipse.jetty.server.Server.handle(Server.java:350)

at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:454)

at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:890)

at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:944)

at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:630)

at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:230)

at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:77)

at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:620)

at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:46)

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:603)

at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:538)

at java.lang.Thread.run(Unknown Source)

I am using the Java service wrapper to wrap ES as a windows service.(https://github.com/elasticsearch/elasticsearch-servicewrapper)

I get this error as soon as the service is started and i am unable to use elastic search till i restart the service a couple of times.

Hope this helps.

Please let me know if you need more information.

regards,

-Parag

[imotov]

I downloaded https://github.com/elasticsearch/elasticsearch-servicewrapper and it runs fine from the command line. However, I am having difficulties installing it as a service. Any pointers on how to do it?

[Parag Dere]

Hi Imotov,

I agree it runs from command prompt without any issues. It is only when ES is configured as a windows service that we get this error.

Please follow the following steps to install ES as a servic:

You need to copy the service folder from the zip file into c:\Elasticsearch\bin (assuming your ES is installed in C:\ElasticSearch)

Ensure the ES_Home variable is set to C:\ElasticSearch\ - dont miss on the trailing slash

Open command prompt and change directory to C:\ElasticSearch\Bin\service\exec

Run the command elasticsearch-windows-x86-32.exec   -i   “C:\ElasticsearchServicewrapper\service\elasticsearch.conf”

This should install the service. Then use some local credentials to configure the service and run it.

If i disable Jetty, the service runs just perfect everytime. On enabling Jetty plugin, i start getting the error mentioned intermittently.

 

regards,

-Parag

[Parag Dere]

Hi Imotov,

Today i configured ES as a service using Apache Commons Daemon (some excellent documentation here http://captaincodeman.com/2011/05/20/elasticsearch-windows-service-2008-x64/)

And ES works perfectly fine with   Jetty... No issues whatsoever.

So it definitely looks like an issue on the Elasticsearch service wrapper.

Not sure if you would still like to investigate or i can log a bug for ESServiceWrapper which coincidentally is managed by Kimchy.

 

regards,

-Parag

[imotov]

Hi Parag,

Thanks for the update and help with reproducing the issue. I agree with you that it looks like a class loader issue in ESServiceWrapper. At least, it is not limited to elasticsearch-jetty. I was able to reproduce it with ICU analysis plugin for example. Basically, any plugin that tries to load classes from plugin classpath on non-bootstrap threads fail on Windows. 

I am glad to hear that you found a solution that works for you. A possible workaround for ESServiceWrapper is explicitly specifying jetty plugin in the class path. It can be done by adding this line to elasticsearch.conf:

wrapper.java.classpath.5=%ES_HOME%/plugins/jetty/*.jar

Thanks you,

Igor

[Parag Dere]

Hi Igor,

Thanks a lot for the response.

I would prefer to use the ESServicewrapper, as i remember reading somewhere that the wrapper also does a check on the server and if the server stops responding, it restarts the service.

The workaround you mentioned works perfectly fine !!

Thanks a lot for all the help

regards,

-Parag