Janus graph stops responding after script evaluation timeout

[Amit Chandak]

Hi, 

       Not sure if this is a gremlin issue, per say, still posting, as looking for some guidance here.

      I am running Janus 0.3.1 with berkeleyJE as the storage backend. I am running Janus as a systemd service. When there is a script evaluatiion timeout, janus stops responding

Oct 02 19:19:14 ip-192-168-28-141.us-west-2.compute.internal bash[9821]: 7013 [main] INFO  org.apache.tinkerpop.gremlin.server.AbstractChannelizer  - application/json already has org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSe

Oct 02 19:19:14 ip-192-168-28-141.us-west-2.compute.internal bash[9821]: 7058 [gremlin-server-boss-1] INFO  org.apache.tinkerpop.gremlin.server.GremlinServer  - Gremlin Server configured with worker thread pool of 16, gremlin pool of 64

Oct 02 19:19:14 ip-192-168-28-141.us-west-2.compute.internal bash[9821]: 7058 [gremlin-server-boss-1] INFO  org.apache.tinkerpop.gremlin.server.GremlinServer  - Channel started at port 8182.

Oct 02 19:32:02 ip-192-168-28-141.us-west-2.compute.internal bash[9821]: 774313 [gremlin-server-exec-28] WARN  org.janusgraph.graphdb.transaction.StandardJanusGraphTx  - Query requires iterating over all vertices [()]. For better perfor

Oct 02 19:37:02 ip-192-168-28-141.us-west-2.compute.internal bash[9821]: 1074402 [gremlin-server-worker-13] WARN  org.apache.tinkerpop.gremlin.server.op.AbstractEvalOpProcessor  - Script evaluation exceeded the configured threshold for

Oct 02 19:37:02 ip-192-168-28-141.us-west-2.compute.internal bash[9821]: java.util.concurrent.TimeoutException: Script evaluation exceeded the configured 'scriptEvaluationTimeout' threshold of 300000 ms or evaluation was otherwise cance

Oct 02 19:37:02 ip-192-168-28-141.us-west-2.compute.internal bash[9821]: at org.apache.tinkerpop.gremlin.groovy.engine.GremlinExecutor.lambda$eval$1(GremlinExecutor.java:310)

Oct 02 19:37:02 ip-192-168-28-141.us-west-2.compute.internal bash[9821]: at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38)

Oct 02 19:37:02 ip-192-168-28-141.us-west-2.compute.internal bash[9821]: at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:120)

Oct 02 19:37:02 ip-192-168-28-141.us-west-2.compute.internal bash[9821]: at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:399)

Oct 02 19:37:02 ip-192-168-28-141.us-west-2.compute.internal bash[9821]: at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:464)

Oct 02 19:37:02 ip-192-168-28-141.us-west-2.compute.internal bash[9821]: at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)

Oct 02 19:37:02 ip-192-168-28-141.us-west-2.compute.internal bash[9821]: at java.lang.Thread.run(Thread.java:748)

Oct 02 19:37:02 ip-192-168-28-141.us-west-2.compute.internal bash[9821]: 1074429 [gremlin-server-exec-29] WARN  org.janusgraph.graphdb.transaction.StandardJanusGraphTx  - Query requires iterating over all vertices [()]. For better perfo

Oct 02 19:37:02 ip-192-168-28-141.us-west-2.compute.internal bash[9821]: 1074431 [gremlin-server-exec-29] WARN  org.apache.tinkerpop.gremlin.server.op.AbstractEvalOpProcessor  - Script evaluation exceeded the configured threshold for re

Oct 02 19:37:02 ip-192-168-28-141.us-west-2.compute.internal bash[9821]: java.util.concurrent.TimeoutException: Script evaluation exceeded the configured 'scriptEvaluationTimeout' threshold of 300000 ms or evaluation was otherwise cance

Oct 02 19:37:02 ip-192-168-28-141.us-west-2.compute.internal bash[9821]: at org.apache.tinkerpop.gremlin.groovy.engine.GremlinExecutor.lambda$eval$0(GremlinExecutor.java:291)

Oct 02 19:37:02 ip-192-168-28-141.us-west-2.compute.internal bash[9821]: at java.util.concurrent.FutureTask.run(FutureTask.java:266)

Oct 02 19:37:02 ip-192-168-28-141.us-west-2.compute.internal bash[9821]: at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

Oct 02 19:37:02 ip-192-168-28-141.us-west-2.compute.internal bash[9821]: at java.util.concurrent.FutureTask.run(FutureTask.java:266)

Oct 02 19:37:02 ip-192-168-28-141.us-west-2.compute.internal bash[9821]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

Oct 02 19:37:02 ip-192-168-28-141.us-west-2.compute.internal bash[9821]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

Oct 02 19:37:02 ip-192-168-28-141.us-west-2.compute.internal bash[9821]: at java.lang.Thread.run(Thread.java:748)

Timeouts can happen bcos of various reasons, but dont understand why Janus stops responding to follow up queries?

Here is my systemd file

[Unit]

Description=JanusGraph

Documentation=http://janusgraph.org/

After=network.target

[Service]

Type=simple

Environment='JAVA_OPTIONS=-Xms512m -Xmx131072m'

WorkingDirectory=/home/ec2-user/janusgraph-0.3.1-hadoop2/

# Using /bin/bash gets around potential SELinux issues with Java

ExecStart=/bin/bash -c '/home/ec2-user/janusgraph-0.3.1-hadoop2/bin/gremlin-server.sh /home/ec2-user/janusgraph-0.3.1-hadoop2/conf/gremlin-server/gremlin-server.yaml'

ExecStop=/bin/bash -c '/home/ec2-user/janusgraph-0.3.1-hadoop2/bin/janusgraph.sh stop'

[Install]

WantedBy=multi-user.target

Thanks

Amit

[Stephen Mallette]

Does a single timeout trigger this problem? Or do you have a series of timeouts that eventually stop it from responding to future requests? What are the failures of the future requests both for the client and server?

To try to isolate the problem, perhaps you could simplify your environment by trying to get a failure with Gremlin Server by itself using TinkerGraph? If you did that, then we'd know it had nothing to do with the underlying system?

--
You received this message because you are subscribed to the Google Groups "Gremlin-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gremlin-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/8f874bd6-5042-4b72-9698-7ad3fb5fff88%40googlegroups.com.

[Amit Chandak]

Thanks Stephen, let me try what u suggested.

On Thursday, October 3, 2019 at 3:02:07 AM UTC-7, Stephen Mallette wrote:

Does a single timeout trigger this problem? Or do you have a series of timeouts that eventually stop it from responding to future requests? What are the failures of the future requests both for the client and server?

To try to isolate the problem, perhaps you could simplify your environment by trying to get a failure with Gremlin Server by itself using TinkerGraph? If you did that, then we'd know it had nothing to do with the underlying system?

To unsubscribe from this group and stop receiving emails from it, send an email to gremli...@googlegroups.com.