Rexster enters bad state

110 views
Skip to first unread message

Farzad Towhidi

unread,
Sep 17, 2014, 5:02:12 PM9/17/14
to aureliu...@googlegroups.com
This one is really hard to explain but sometimes Rexster enters a bad state where it cannot process anymore requests. After which it either returns 
<html><body><h1>Internal Error</h1></body></html>
or 
An error occurred while generating the response object

Also, when I try to access "localhost:8182/doghouse" to see if I can play around with the gremlin console, the page never loads - At this point, I am forced to restart Rexster. Wish I could give more information on what the actual cause is but it happens at random.

Has anyone else encountered this problem? If so, is there a workaround?

Thanks,
Farzad


Stephen Mallette

unread,
Sep 17, 2014, 6:56:05 PM9/17/14
to aureliu...@googlegroups.com
I'm just guessing here, but if it happens under many concurrent requests or you're on certain brands of Linux (Red Hat comes to mind) you might need to increase your thread pool settings....here's the classic issue in Rexster - good ol' 271:


--
You received this message because you are subscribed to the Google Groups "Aurelius" group.
To unsubscribe from this group and stop receiving emails from it, send an email to aureliusgraph...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/aureliusgraphs/aa9de24c-44ea-4ccc-b142-57b13e51a425%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Farzad Towhidi

unread,
Sep 22, 2014, 10:51:10 AM9/22/14
to aureliu...@googlegroups.com
Will this just hide the problem or actually fix it? 

Stephen Mallette

unread,
Sep 22, 2014, 11:42:13 AM9/22/14
to aureliu...@googlegroups.com
The default rexster.xml comes configured with a pretty small thread pool.  I'd say that in a production environment it's probably a bad idea to run with those default settings.  It just so happens that for some reason RedHat requires larger settings that most.  No one I know of running in that environment has ever complained of problems after doing that.

To view this discussion on the web visit https://groups.google.com/d/msgid/aureliusgraphs/26887052-c777-4a02-a10a-a5222f7c44ec%40googlegroups.com.

Farzad Towhidi

unread,
Oct 6, 2014, 4:47:19 PM10/6/14
to aureliu...@googlegroups.com
Thanks for the suggestion but unfortunately changing thread pool size did not do the trick. I'm running it on MacOSX if that makes any difference.

Stephen Mallette

unread,
Oct 6, 2014, 4:51:21 PM10/6/14
to aureliu...@googlegroups.com
Do any of the logs show errors on the way to "freezing"? Short of that, you may need to try to give us some reproduction steps that allow us to see the problem or some other description of how you are doing things to get Rexster to this state.  

--
You received this message because you are subscribed to the Google Groups "Aurelius" group.
To unsubscribe from this group and stop receiving emails from it, send an email to aureliusgraph...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/aureliusgraphs/a653688f-dbf8-4c1c-ab53-1f9486115e17%40googlegroups.com.

Farzad Towhidi

unread,
Oct 6, 2014, 5:19:18 PM10/6/14
to aureliu...@googlegroups.com
Ok I see two errors here:
2130837 [Grizzly(1)] ERROR com.tinkerpop.rexster.GraphResource  - Dynamic invocation of the [tp:gremlin+*] extension failed.
java.lang.reflect.InvocationTargetException

and 

2131770 [Grizzly(32)] ERROR com.tinkerpop.rexster.GraphResource  - It would be smart to trap this this exception within the extension and supply a good response to the user:PermGen space

java.lang.OutOfMemoryError: PermGen space



Full stack trace:


2130837 [Grizzly(1)] ERROR com.tinkerpop.rexster.GraphResource  - Dynamic invocation of the [tp:gremlin+*] extension failed.







java.lang.reflect.InvocationTargetException


        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)


        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)


        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)


        at java.lang.reflect.Method.invoke(Method.java:606)


        at com.tinkerpop.rexster.AbstractSubResource.invokeExtension(AbstractSubResource.java:322)


        at com.tinkerpop.rexster.AbstractSubResource.invokeExtension(AbstractSubResource.java:229)


        at com.tinkerpop.rexster.GraphResource.executeGraphExtension(GraphResource.java:281)


        at com.tinkerpop.rexster.GraphResource.postGraphExtension(GraphResource.java:201)


        at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)


        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)


        at java.lang.reflect.Method.invoke(Method.java:606)


        at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)


        at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)


        at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)


        at com.codahale.metrics.jersey.InstrumentedResourceMethodDispatchProvider$TimedRequestDispatcher.dispatch(InstrumentedResourceMethodDispatchProvider.java:30)


        at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)


        at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)


        at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)


        at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)


        at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)


        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1511)


        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1442)


        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1391)


        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1381)


        at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)


        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:538)


        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:716)


        at javax.servlet.http.HttpServlet.service(HttpServlet.java:770)


        at org.glassfish.grizzly.servlet.FilterChainImpl.doFilter(FilterChainImpl.java:147)


        at org.glassfish.grizzly.servlet.FilterChainImpl.invokeFilterChain(FilterChainImpl.java:106)


        at org.glassfish.grizzly.servlet.ServletHandler.doServletService(ServletHandler.java:252)


        at org.glassfish.grizzly.servlet.ServletHandler.service(ServletHandler.java:188)


        at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:164)


        at org.glassfish.grizzly.http.server.HttpHandlerChain.service(HttpHandlerChain.java:196)


        at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:164)


        at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:175)


        at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)


        at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:265)


        at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:200)


        at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:134)


        at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112)


        at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:78)


        at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:815)


        at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112)


        at org.glassfish.grizzly.strategies.LeaderFollowerNIOStrategy.executeIoEvent(LeaderFollowerNIOStrategy.java:102)


        at org.glassfish.grizzly.strategies.AbstractIOStrategy.executeIoEvent(AbstractIOStrategy.java:88)


        at org.glassfish.grizzly.nio.SelectorRunner.iterateKeyEvents(SelectorRunner.java:398)


        at org.glassfish.grizzly.nio.SelectorRunner.iterateKeys(SelectorRunner.java:368)


        at org.glassfish.grizzly.nio.SelectorRunner.doSelect(SelectorRunner.java:334)


        at org.glassfish.grizzly.nio.SelectorRunner.run(SelectorRunner.java:264)


        at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:567)


        at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:547)


        at java.lang.Thread.run(Thread.java:744)


Caused by: java.lang.OutOfMemoryError: PermGen space


        at java.lang.ClassLoader.defineClass1(Native Method)


        at java.lang.ClassLoader.defineClass(ClassLoader.java:800)


        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)


        at groovy.lang.GroovyClassLoader.access$300(GroovyClassLoader.java:55)


        at groovy.lang.GroovyClassLoader$ClassCollector.createClass(GroovyClassLoader.java:515)


        at groovy.lang.GroovyClassLoader$ClassCollector.onClassNode(GroovyClassLoader.java:532)


        at groovy.lang.GroovyClassLoader$ClassCollector.call(GroovyClassLoader.java:536)


        at org.codehaus.groovy.control.CompilationUnit$14.call(CompilationUnit.java:779)


        at org.codehaus.groovy.control.CompilationUnit$14.call(CompilationUnit.java:787)


        at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:970)


        at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:548)


        at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:526)


        at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:503)


        at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:302)


        at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:281)


        at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:267)


        at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:263)


        at com.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine.getScriptClass(GremlinGroovyScriptEngine.java:231)


        at com.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine.eval(GremlinGroovyScriptEngine.java:90)


        at javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:233)


        at com.tinkerpop.rexster.gremlin.GremlinExtension.tryExecuteGremlinScript(GremlinExtension.java:275)


        at com.tinkerpop.rexster.gremlin.GremlinExtension.evaluatePostOnGraph(GremlinExtension.java:193)


        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)


        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)


        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)


        at java.lang.reflect.Method.invoke(Method.java:606)


        at com.tinkerpop.rexster.AbstractSubResource.invokeExtension(AbstractSubResource.java:322)


        at com.tinkerpop.rexster.AbstractSubResource.invokeExtension(AbstractSubResource.java:229)


        at com.tinkerpop.rexster.GraphResource.executeGraphExtension(GraphResource.java:281)


        at com.tinkerpop.rexster.GraphResource.postGraphExtension(GraphResource.java:201)


        at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)


        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)




On Wednesday, September 17, 2014 5:02:12 PM UTC-4, Farzad Towhidi wrote:

Stephen Mallette

unread,
Oct 7, 2014, 6:13:49 AM10/7/14
to aureliu...@googlegroups.com
I guess you could increase permgen space to solve your problem, but what kind of requests are you issuing to get this error?

--
You received this message because you are subscribed to the Google Groups "Aurelius" group.
To unsubscribe from this group and stop receiving emails from it, send an email to aureliusgraph...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/aureliusgraphs/e52eae64-b05a-423c-90ed-8f07d17f3c16%40googlegroups.com.

Farzad Towhidi

unread,
Oct 7, 2014, 10:27:17 AM10/7/14
to aureliu...@googlegroups.com
The gremlin scripts that I am building are quite large. Especially when creating multiple nodes/edges with lots of properties simultaneously, however, the script length never surpasses the maximum amount (2M). What are the scenarios that could result in such an error?


On Wednesday, September 17, 2014 5:02:12 PM UTC-4, Farzad Towhidi wrote:

Stephen Mallette

unread,
Oct 7, 2014, 10:39:40 AM10/7/14
to aureliu...@googlegroups.com
If you are submitting those scripts over and over again and not parameterizing requests then you could run into this issue.  The following is for RexPro but applies equally to the Gremlin Extension:


--
You received this message because you are subscribed to the Google Groups "Aurelius" group.
To unsubscribe from this group and stop receiving emails from it, send an email to aureliusgraph...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/aureliusgraphs/ebd70bd9-4312-4cb5-bdfe-8ff83245ef80%40googlegroups.com.

Farzad Towhidi

unread,
Oct 7, 2014, 10:49:15 AM10/7/14
to aureliu...@googlegroups.com
Thanks for the reply. I will try that. I also found an interesting article that explains the PermGen situation:
https://plumbr.eu/outofmemoryerror/permgen-space#solution

It suggests to add the following flags when launching the application:
-XX:MaxPermSize=512m
-XX:+CMSClassUnloadingEnabled
-XX:+UseConcMarkSweepGC

Would any of these flags cause any problems?


On Wednesday, September 17, 2014 5:02:12 PM UTC-4, Farzad Towhidi wrote:

Stephen Mallette

unread,
Oct 7, 2014, 10:57:32 AM10/7/14
to aureliu...@googlegroups.com
That's what i was essentially suggesting earlier - increase the permgen space.  I think a longer term solution however should be to figure out how to reduce class creation given the scripts your sending to the server.

--
You received this message because you are subscribed to the Google Groups "Aurelius" group.
To unsubscribe from this group and stop receiving emails from it, send an email to aureliusgraph...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/aureliusgraphs/2cc677fc-74e1-4158-89c9-81f92f569a1b%40googlegroups.com.

Farzad Towhidi

unread,
Oct 9, 2014, 11:03:41 AM10/9/14
to aureliu...@googlegroups.com
Thanks alot for your help, parametrizing my queries did the trick! It also improved performance significantly, queries that took 700ms to commit now take ~10ms!


On Wednesday, September 17, 2014 5:02:12 PM UTC-4, Farzad Towhidi wrote:
Reply all
Reply to author
Forward
0 new messages