Rexster enters bad state

瀏覽次數：110 次
跳到第一則未讀訊息

Farzad Towhidi

未讀,
2014年9月17日 下午5:02:122014/9/17
收件者：aureliu...@googlegroups.com
This one is really hard to explain but sometimes Rexster enters a bad state where it cannot process anymore requests. After which it either returns 
<html><body><h1>Internal Error</h1></body></html>
or 
An error occurred while generating the response object

Also, when I try to access "localhost:8182/doghouse" to see if I can play around with the gremlin console, the page never loads - At this point, I am forced to restart Rexster. Wish I could give more information on what the actual cause is but it happens at random.

Has anyone else encountered this problem? If so, is there a workaround?

Thanks,
Farzad


Stephen Mallette

未讀,
2014年9月17日 下午6:56:052014/9/17
收件者：aureliu...@googlegroups.com
I'm just guessing here, but if it happens under many concurrent requests or you're on certain brands of Linux (Red Hat comes to mind) you might need to increase your thread pool settings....here's the classic issue in Rexster - good ol' 271:


--
You received this message because you are subscribed to the Google Groups "Aurelius" group.
To unsubscribe from this group and stop receiving emails from it, send an email to aureliusgraph...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/aureliusgraphs/aa9de24c-44ea-4ccc-b142-57b13e51a425%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Farzad Towhidi

未讀,
2014年9月22日 上午10:51:102014/9/22
收件者：aureliu...@googlegroups.com
Will this just hide the problem or actually fix it? 

Stephen Mallette

未讀,
2014年9月22日 上午11:42:132014/9/22
收件者：aureliu...@googlegroups.com
The default rexster.xml comes configured with a pretty small thread pool.  I'd say that in a production environment it's probably a bad idea to run with those default settings.  It just so happens that for some reason RedHat requires larger settings that most.  No one I know of running in that environment has ever complained of problems after doing that.

To view this discussion on the web visit https://groups.google.com/d/msgid/aureliusgraphs/26887052-c777-4a02-a10a-a5222f7c44ec%40googlegroups.com.

Farzad Towhidi

未讀,
2014年10月6日 下午4:47:192014/10/6
收件者：aureliu...@googlegroups.com
Thanks for the suggestion but unfortunately changing thread pool size did not do the trick. I'm running it on MacOSX if that makes any difference.

Stephen Mallette

未讀,
2014年10月6日 下午4:51:212014/10/6
收件者：aureliu...@googlegroups.com
Do any of the logs show errors on the way to "freezing"? Short of that, you may need to try to give us some reproduction steps that allow us to see the problem or some other description of how you are doing things to get Rexster to this state.  

--
You received this message because you are subscribed to the Google Groups "Aurelius" group.
To unsubscribe from this group and stop receiving emails from it, send an email to aureliusgraph...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/aureliusgraphs/a653688f-dbf8-4c1c-ab53-1f9486115e17%40googlegroups.com.

Farzad Towhidi

未讀,
2014年10月6日 下午5:19:182014/10/6
收件者：aureliu...@googlegroups.com
Ok I see two errors here:
2130837 [Grizzly(1)] ERROR com.tinkerpop.rexster.GraphResource  - Dynamic invocation of the [tp:gremlin+*] extension failed.
java.lang.reflect.InvocationTargetException

and 

2131770 [Grizzly(32)] ERROR com.tinkerpop.rexster.GraphResource  - It would be smart to trap this this exception within the extension and supply a good response to the user:PermGen space

java.lang.OutOfMemoryError: PermGen space



Full stack trace:


2130837 [Grizzly(1)] ERROR com.tinkerpop.rexster.GraphResource  - Dynamic invocation of the [tp:gremlin+*] extension failed.







java.lang.reflect.InvocationTargetException


        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)


        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)


        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)


        at java.lang.reflect.Method.invoke(Method.java:606)


        at com.tinkerpop.rexster.AbstractSubResource.invokeExtension(AbstractSubResource.java:322)


        at com.tinkerpop.rexster.AbstractSubResource.invokeExtension(AbstractSubResource.java:229)


        at com.tinkerpop.rexster.GraphResource.executeGraphExtension(GraphResource.java:281)


        at com.tinkerpop.rexster.GraphResource.postGraphExtension(GraphResource.java:201)


        at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)


        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)


        at java.lang.reflect.Method.invoke(Method.java:606)


        at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)


        at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)


        at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)


        at com.codahale.metrics.jersey.InstrumentedResourceMethodDispatchProvider$TimedRequestDispatcher.dispatch(InstrumentedResourceMethodDispatchProvider.java:30)


        at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)


        at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)


        at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)


        at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)


        at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)


        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1511)


        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1442)


        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1391)


        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1381)


        at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)


        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:538)


        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:716)


        at javax.servlet.http.HttpServlet.service(HttpServlet.java:770)


        at org.glassfish.grizzly.servlet.FilterChainImpl.doFilter(FilterChainImpl.java:147)


        at org.glassfish.grizzly.servlet.FilterChainImpl.invokeFilterChain(FilterChainImpl.java:106)


        at org.glassfish.grizzly.servlet.ServletHandler.doServletService(ServletHandler.java:252)


        at org.glassfish.grizzly.servlet.ServletHandler.service(ServletHandler.java:188)


        at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:164)


        at org.glassfish.grizzly.http.server.HttpHandlerChain.service(HttpHandlerChain.java:196)


        at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:164)


        at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:175)


        at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)


        at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:265)


        at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:200)


        at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:134)


        at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112)


        at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:78)


        at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:815)


        at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112)


        at org.glassfish.grizzly.strategies.LeaderFollowerNIOStrategy.executeIoEvent(LeaderFollowerNIOStrategy.java:102)


        at org.glassfish.grizzly.strategies.AbstractIOStrategy.executeIoEvent(AbstractIOStrategy.java:88)


        at org.glassfish.grizzly.nio.SelectorRunner.iterateKeyEvents(SelectorRunner.java:398)


        at org.glassfish.grizzly.nio.SelectorRunner.iterateKeys(SelectorRunner.java:368)


        at org.glassfish.grizzly.nio.SelectorRunner.doSelect(SelectorRunner.java:334)


        at org.glassfish.grizzly.nio.SelectorRunner.run(SelectorRunner.java:264)


        at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:567)


        at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:547)


        at java.lang.Thread.run(Thread.java:744)


Caused by: java.lang.OutOfMemoryError: PermGen space


        at java.lang.ClassLoader.defineClass1(Native Method)


        at java.lang.ClassLoader.defineClass(ClassLoader.java:800)


        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)


        at groovy.lang.GroovyClassLoader.access$300(GroovyClassLoader.java:55)


        at groovy.lang.GroovyClassLoader$ClassCollector.createClass(GroovyClassLoader.java:515)


        at groovy.lang.GroovyClassLoader$ClassCollector.onClassNode(GroovyClassLoader.java:532)


        at groovy.lang.GroovyClassLoader$ClassCollector.call(GroovyClassLoader.java:536)


        at org.codehaus.groovy.control.CompilationUnit$14.call(CompilationUnit.java:779)


        at org.codehaus.groovy.control.CompilationUnit$14.call(CompilationUnit.java:787)


        at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:970)


        at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:548)


        at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:526)


        at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:503)


        at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:302)


        at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:281)


        at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:267)


        at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:263)


        at com.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine.getScriptClass(GremlinGroovyScriptEngine.java:231)


        at com.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine.eval(GremlinGroovyScriptEngine.java:90)


        at javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:233)


        at com.tinkerpop.rexster.gremlin.GremlinExtension.tryExecuteGremlinScript(GremlinExtension.java:275)


        at com.tinkerpop.rexster.gremlin.GremlinExtension.evaluatePostOnGraph(GremlinExtension.java:193)


        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)


        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)


        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)


        at java.lang.reflect.Method.invoke(Method.java:606)


        at com.tinkerpop.rexster.AbstractSubResource.invokeExtension(AbstractSubResource.java:322)


        at com.tinkerpop.rexster.AbstractSubResource.invokeExtension(AbstractSubResource.java:229)


        at com.tinkerpop.rexster.GraphResource.executeGraphExtension(GraphResource.java:281)


        at com.tinkerpop.rexster.GraphResource.postGraphExtension(GraphResource.java:201)


        at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)


        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)




On Wednesday, September 17, 2014 5:02:12 PM UTC-4, Farzad Towhidi wrote:

Stephen Mallette

未讀,
2014年10月7日 清晨6:13:492014/10/7
收件者：aureliu...@googlegroups.com
I guess you could increase permgen space to solve your problem, but what kind of requests are you issuing to get this error?

--
You received this message because you are subscribed to the Google Groups "Aurelius" group.
To unsubscribe from this group and stop receiving emails from it, send an email to aureliusgraph...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/aureliusgraphs/e52eae64-b05a-423c-90ed-8f07d17f3c16%40googlegroups.com.

Farzad Towhidi

未讀,
2014年10月7日 上午10:27:172014/10/7
收件者：aureliu...@googlegroups.com
The gremlin scripts that I am building are quite large. Especially when creating multiple nodes/edges with lots of properties simultaneously, however, the script length never surpasses the maximum amount (2M). What are the scenarios that could result in such an error?


On Wednesday, September 17, 2014 5:02:12 PM UTC-4, Farzad Towhidi wrote:

Stephen Mallette

未讀,
2014年10月7日 上午10:39:402014/10/7
收件者：aureliu...@googlegroups.com
If you are submitting those scripts over and over again and not parameterizing requests then you could run into this issue.  The following is for RexPro but applies equally to the Gremlin Extension:


--
You received this message because you are subscribed to the Google Groups "Aurelius" group.
To unsubscribe from this group and stop receiving emails from it, send an email to aureliusgraph...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/aureliusgraphs/ebd70bd9-4312-4cb5-bdfe-8ff83245ef80%40googlegroups.com.

Farzad Towhidi

未讀,
2014年10月7日 上午10:49:152014/10/7
收件者：aureliu...@googlegroups.com
Thanks for the reply. I will try that. I also found an interesting article that explains the PermGen situation:
https://plumbr.eu/outofmemoryerror/permgen-space#solution

It suggests to add the following flags when launching the application:
-XX:MaxPermSize=512m
-XX:+CMSClassUnloadingEnabled
-XX:+UseConcMarkSweepGC

Would any of these flags cause any problems?


On Wednesday, September 17, 2014 5:02:12 PM UTC-4, Farzad Towhidi wrote:

Stephen Mallette

未讀,
2014年10月7日 上午10:57:322014/10/7
收件者：aureliu...@googlegroups.com
That's what i was essentially suggesting earlier - increase the permgen space.  I think a longer term solution however should be to figure out how to reduce class creation given the scripts your sending to the server.

--
You received this message because you are subscribed to the Google Groups "Aurelius" group.
To unsubscribe from this group and stop receiving emails from it, send an email to aureliusgraph...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/aureliusgraphs/2cc677fc-74e1-4158-89c9-81f92f569a1b%40googlegroups.com.

Farzad Towhidi

未讀,
2014年10月9日 上午11:03:412014/10/9
收件者：aureliu...@googlegroups.com
Thanks alot for your help, parametrizing my queries did the trick! It also improved performance significantly, queries that took 700ms to commit now take ~10ms!


On Wednesday, September 17, 2014 5:02:12 PM UTC-4, Farzad Towhidi wrote:
回覆所有人
回覆作者
轉寄
0 則新訊息