Error while Shutting Down Gremlin Server

Sarthak Ghosh

unread,

Mar 24, 2020, 4:28:08 PM3/24/20

to Gremlin-users

Hi,

While stopping a gremlin-server, I'm seeing below lines in the log

```
20:23:08.959 [gremlin-server-shutdown] INFO o.a.t.gremlin.server.op.OpLoader - Adding the cypher OpProcessor.
20:23:08.963 [gremlin-server-shutdown] INFO o.a.t.gremlin.server.op.OpLoader - Adding the standard OpProcessor.
20:23:08.968 [gremlin-server-shutdown] INFO o.a.t.gremlin.server.op.OpLoader - Adding the session OpProcessor.
20:23:09.163 [gremlin-server-shutdown] INFO o.a.t.gremlin.server.op.OpLoader - Adding the traversal OpProcessor.

20:01:51.200 [gremlin-server-shutdown] INFO o.a.t.gremlin.server.GremlinServer - Shutting down OpProcessor[]

20:01:51.201 [gremlin-server-shutdown] INFO o.a.t.gremlin.server.GremlinServer - Shutting down OpProcessor[cypher]

20:01:51.201 [gremlin-server-shutdown] INFO o.a.t.gremlin.server.GremlinServer - Shutting down OpProcessor[session]

20:01:51.202 [gremlin-server-shutdown] INFO o.a.t.gremlin.server.GremlinServer - Shutting down OpProcessor[traversal]

20:01:51.203 [gremlin-server-shutdown] INFO o.a.t.gremlin.server.GremlinServer - Shutting down thread pools.

20:01:51.203 [gremlin-server-shutdown] DEBUG o.a.t.gremlin.server.GremlinServer - Shutdown Gremlin thread pool.

Exception in thread "gremlin-server-shutdown" java.lang.NullPointerException

at org.apache.tinkerpop.gremlin.server.GremlinServer.stop(GremlinServer.java:255)

at org.apache.tinkerpop.gremlin.server.GremlinServer.lambda$new$0(GremlinServer.java:103)

at java.lang.Thread.run(Thread.java:748)

```

Version 3.4.4

Stop command:
cd apache-tinkerpop-gremlin-server-3.4.4/bin
./gremlin.sh stop

In a normal scenario, this should be the o/p

```

12:57:08.661 [gremlin-server-shutdown] INFO o.a.t.gremlin.server.GremlinServer - Shutting down OpProcessor[]

12:57:08.661 [gremlin-server-shutdown] INFO o.a.t.gremlin.server.GremlinServer - Shutting down OpProcessor[session]

12:57:08.663 [gremlin-server-shutdown] INFO o.a.t.gremlin.server.GremlinServer - Shutting down OpProcessor[traversal]

12:57:08.664 [gremlin-server-shutdown] INFO o.a.t.gremlin.server.GremlinServer - Shutting down thread pools.

12:57:08.664 [gremlin-server-shutdown] DEBUG o.a.t.gremlin.server.GremlinServer - Shutdown Gremlin thread pool.

12:57:08.665 [gremlin-server-shutdown] DEBUG o.a.t.gremlin.server.GremlinServer - Shutdown Worker thread pool.

12:57:08.666 [gremlin-server-shutdown] DEBUG o.a.t.gremlin.server.GremlinServer - Shutdown Boss thread pool.

12:57:10.874 [gremlin-server-stop] INFO o.a.t.gremlin.server.GremlinServer - Gremlin Server - shutdown complete
```

Any particular reason why this is causing an issue and any way to solve this?

Stephen Mallette

unread,

Mar 24, 2020, 4:55:00 PM3/24/20

to gremli...@googlegroups.com

And there are no other errors in your logs? It's only that error on shutdown? If so, that's strange because I can't think of a state that Gremlin Server would get into where it's ExecutorService can remain null, unless there is a failure to create the ExecutorService in the first place but then I'd imagine other errors would ensue.

--
You received this message because you are subscribed to the Google Groups "Gremlin-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gremlin-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/6dd50216-ba24-414f-bc8e-4eb59db66b75%40googlegroups.com.

Sarthak Ghosh

unread,

Mar 24, 2020, 5:00:17 PM3/24/20

to Gremlin-users

Hi Stephen,
There are other errors as well. I'm connecting to a HDP machine. Sometimes, when my HDP is down, graphs mentioned in gremlin-server.yaml are not able to load so I get the following error
```

java.net.ConnectException: Connection refused

at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.8.0_242]

at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:714) ~[na:1.8.0_242]

at org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361) ~[hbase-shaded-client-2.1.5.jar:2.1.5]

at org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141) ~[hbase-shaded-client-2.1.5.jar:2.1.5]

20:51:35.831 [ReadOnlyZKClient-pushkar-hdp2.fci-dev.com:2181,pushkar-hdp3.fci-dev.com:2181@0x5471388b-SendThread(pushkar-hdp2.fci-dev.com:2181)] DEBUG o.a.h.h.s.o.a.z.ClientCnxnSocketNIO - Ignoring exception during shutdown input

java.nio.channels.ClosedChannelException: null

at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:777) ~[na:1.8.0_242]

at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:412) ~[na:1.8.0_242]

at org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:200) ~[hbase-shaded-client-2.1.5.jar:2.1.5]

at org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1246) [hbase-shaded-client-2.1.5.jar:2.1.5]

at org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1170) [hbase-shaded-client-2.1.5.jar:2.1.5]

20:51:35.831 [ReadOnlyZKClient-pushkar-hdp2.fci-dev.com:2181,pushkar-hdp3.fci-dev.com:2181@0x5471388b-SendThread(pushkar-hdp2.fci-dev.com:2181)] DEBUG o.a.h.h.s.o.a.z.ClientCnxnSocketNIO - Ignoring exception during shutdown output

java.nio.channels.ClosedChannelException: null

at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:794) ~[na:1.8.0_242]

at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:420) ~[na:1.8.0_242]

at org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:207) ~[hbase-shaded-client-2.1.5.jar:2.1.5]

at org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1246) [hbase-shaded-client-2.1.5.jar:2.1.5]

at org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1170) [hbase-shaded-client-2.1.5.jar:2.1.5]
```

But that is not my concern. My main problem is gremlin-server is not having a clean Shutdown
This log is not seen `12:57:10.874 [gremlin-server-stop] INFO o.a.t.gremlin.server.GremlinServer - Gremlin Server - shutdown complete`

Due this issue, I sometimes get the following error
```

Caused by: org.janusgraph.core.JanusGraphException: A JanusGraph graph with the same instance id [ac1eb8ff42-fci-graph-writer-gremlin-7bc5d757bd-bkr7f1] is already open. Might required forced shutdown.
```

Stephen Mallette

unread,

Mar 24, 2020, 5:06:05 PM3/24/20

to gremli...@googlegroups.com

Well, I guess it's easy to "fix" - we just check for null before calling shutdown():

--

You received this message because you are subscribed to the Google Groups "Gremlin-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gremlin-user...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/98b495b6-150e-4461-a8ba-78955c9fd8f0%40googlegroups.com.

Stephen Mallette

unread,

Mar 24, 2020, 5:08:18 PM3/24/20

to gremli...@googlegroups.com

sorry. hit the send before i was ready...again:

Well, I guess it's easy to "fix" - we just check for null before calling shutdown():

https://github.com/apache/tinkerpop/blob/c3092069f0961759036b55c5f874a4c2fd1d4978/gremlin-server/src/main/java/org/apache/tinkerpop/gremlin/server/GremlinServer.java#L255

However I'd like to know how we get in that state before doing that blindly. I don't see anywhere in the code where we set that value to null and you can't really initialize the GremlinServer class without it being instantiated. That's pretty mysterious. I don't think there's a way to call stop() before gremlinExecutorService is initialized....

anyone have any ideas how this might happen? could it be something with the shell scripting of:

bin/gremlin-server.sh stop

that allows this to happen?

Sarthak Ghosh

unread,

Mar 24, 2020, 5:08:22 PM3/24/20

to Gremlin-users

Is there a work around for this or a forced shutdown?

Sarthak Ghosh

unread,

Mar 24, 2020, 5:23:14 PM3/24/20

to Gremlin-users

I haven't made any changes to the script. They are what came along with the zip.

Below are the logs when the graphs mentioned in gremlin-server.yaml and empty-sample.groovy have successfully loaded

```

21:21:01.387 [gremlin-server-shutdown] INFO o.a.t.gremlin.server.GremlinServer - Shutting down OpProcessor[]

21:21:01.387 [gremlin-server-shutdown] INFO o.a.t.gremlin.server.GremlinServer - Shutting down OpProcessor[cypher]

21:21:01.387 [gremlin-server-shutdown] INFO o.a.t.gremlin.server.GremlinServer - Shutting down OpProcessor[session]

21:21:01.390 [gremlin-server-shutdown] INFO o.a.t.gremlin.server.GremlinServer - Shutting down OpProcessor[traversal]

21:21:01.393 [SIGTERM handler] DEBUG o.j.d.hbase.HBaseStoreManager - Substituted default CF name "systemlog" with short form "m" to reduce HBase KeyValue size

21:21:01.394 [gremlin-server-shutdown] INFO o.a.t.gremlin.server.GremlinServer - Shutting down thread pools.

21:21:01.394 [gremlin-server-shutdown] DEBUG o.a.t.gremlin.server.GremlinServer - Shutdown Gremlin thread pool.

21:21:01.401 [gremlin-server-shutdown] DEBUG o.a.t.gremlin.server.GremlinServer - Shutdown Worker thread pool.

21:21:01.402 [gremlin-server-shutdown] DEBUG o.a.t.gremlin.server.GremlinServer - Shutdown Boss thread pool.

21:21:01.404 [gremlin-server-stop] INFO o.a.t.gremlin.server.GremlinServer - Executing shutdown LifeCycleHook

21:21:01.404 [gremlin-server-stop] INFO o.a.t.gremlin.server.GremlinServer - Executed once at shutdown of Gremlin Server.

21:21:01.407 [SIGTERM handler] DEBUG o.j.d.hbase.HBaseStoreManager - Substituted default CF name "txlog" with short form "l" to reduce HBase KeyValue size

21:21:01.415 [SIGTERM handler] INFO o.a.h.h.c.ConnectionImplementation - Closing master protocol: MasterService

21:21:01.417 [SIGTERM handler] DEBUG o.a.h.h.zookeeper.ReadOnlyZKClient - Close zookeeper connection 0x32130e61 to integ-fcii-auto2.fci-cloud.com:2181,integ-fcii-auto3.fci-cloud.com:2181

21:21:01.418 [SIGTERM handler] DEBUG o.a.h.hbase.ipc.AbstractRpcClient - Stopping rpc client

21:21:01.418 [SIGTERM handler] DEBUG o.a.h.i.n.c.PoolingNHttpClientConnectionManager - Connection manager is shutting down

21:21:01.419 [SIGTERM handler] DEBUG o.a.h.i.n.c.PoolingNHttpClientConnectionManager - Connection manager shut down

21:21:03.606 [gremlin-server-stop] DEBUG o.a.t.gremlin.server.GremlinServer - Closing Graph instance [insights]

21:21:03.607 [gremlin-server-stop] WARN o.j.g.database.StandardJanusGraph - Failed to remove shutdown hook

java.lang.IllegalStateException: Shutdown in progress

at java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:82) ~[na:1.8.0_242]

at java.lang.Runtime.removeShutdownHook(Runtime.java:240) ~[na:1.8.0_242]

at org.janusgraph.graphdb.database.StandardJanusGraph.removeHook(StandardJanusGraph.java:270) [janusgraph-core-0.4.1.jar:na]

at org.janusgraph.graphdb.database.StandardJanusGraph.close(StandardJanusGraph.java:203) [janusgraph-core-0.4.1.jar:na]

at org.apache.tinkerpop.gremlin.server.GremlinServer.lambda$null$7(GremlinServer.java:298) [gremlin-server-3.4.4.jar:3.4.4]

at java.util.concurrent.ConcurrentHashMap$KeySetView.forEach(ConcurrentHashMap.java:4649) ~[na:1.8.0_242]

at org.apache.tinkerpop.gremlin.server.GremlinServer.lambda$stop$8(GremlinServer.java:295) [gremlin-server-3.4.4.jar:3.4.4]

at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_242]

21:21:03.607 [gremlin-server-stop] INFO o.a.t.gremlin.server.GremlinServer - Closed Graph instance [insights]

21:21:03.607 [gremlin-server-stop] DEBUG o.a.t.gremlin.server.GremlinServer - Closing Graph instance [fci]

21:21:03.608 [gremlin-server-stop] WARN o.j.g.database.StandardJanusGraph - Failed to remove shutdown hook

java.lang.IllegalStateException: Shutdown in progress

at java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:82) ~[na:1.8.0_242]

at java.lang.Runtime.removeShutdownHook(Runtime.java:240) ~[na:1.8.0_242]

at org.janusgraph.graphdb.database.StandardJanusGraph.removeHook(StandardJanusGraph.java:270) [janusgraph-core-0.4.1.jar:na]

at org.janusgraph.graphdb.database.StandardJanusGraph.close(StandardJanusGraph.java:203) [janusgraph-core-0.4.1.jar:na]

at org.apache.tinkerpop.gremlin.server.GremlinServer.lambda$null$7(GremlinServer.java:298) [gremlin-server-3.4.4.jar:3.4.4]

at java.util.concurrent.ConcurrentHashMap$KeySetView.forEach(ConcurrentHashMap.java:4649) ~[na:1.8.0_242]

at org.apache.tinkerpop.gremlin.server.GremlinServer.lambda$stop$8(GremlinServer.java:295) [gremlin-server-3.4.4.jar:3.4.4]

at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_242]

21:21:03.608 [gremlin-server-stop] INFO o.a.t.gremlin.server.GremlinServer - Closed Graph instance [fci]

21:21:03.609 [gremlin-server-stop] INFO o.a.t.gremlin.server.GremlinServer - Gremlin Server - shutdown complete
```

So it is when the graphs have not loaded properly, the error comes..

Stephen Mallette

unread,

Mar 24, 2020, 6:09:14 PM3/24/20

to gremli...@googlegroups.com

So, let me try to clarify to make sure I understand - are you saying that when you start Gremlin Server and the graphs fail to load and the server start to self shutdown you get:

20:01:51.203 [gremlin-server-shutdown] DEBUG o.a.t.gremlin.server.GremlinServer - Shutdown Gremlin thread pool.

Exception in thread "gremlin-server-shutdown" java.lang.NullPointerException

at org.apache.tinkerpop.gremlin.server.GremlinServer.stop(GremlinServer.java:255)

at org.apache.tinkerpop.gremlin.server.GremlinServer.lambda$new$0(GremlinServer.java:103)

at java.lang.Thread.run(Thread.java:748)

You don't get this message as a result of calling:

bin/gremlin-server.sh stop

Is that right? If not, could you please provide the reproduction steps as you understand them?

--

You received this message because you are subscribed to the Google Groups "Gremlin-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gremlin-user...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/cb139ed9-e96a-4cb1-9d40-c139d5f96abd%40googlegroups.com.

Sarthak Ghosh

unread,

Mar 24, 2020, 6:16:55 PM3/24/20

to Gremlin-users

The steps are automated since it is inside a kube pod but these are the steps which are carried out
1. server start -> bin/gremlin-server.sh start

2. Check whether graphs mentioned in gremlin-server.yaml have loaded properly or not

3. If failure, then stop the server by calling -> bin/gremlin-server.sh stop
4. Check logs

```
20:23:08.959 [gremlin-server-shutdown] INFO o.a.t.gremlin.server.op.OpLoader - Adding the cypher OpProcessor.
20:23:08.963 [gremlin-server-shutdown] INFO o.a.t.gremlin.server.op.OpLoader - Adding the standard OpProcessor.
20:23:08.968 [gremlin-server-shutdown] INFO o.a.t.gremlin.server.op.OpLoader - Adding the session OpProcessor.
20:23:09.163 [gremlin-server-shutdown] INFO o.a.t.gremlin.server.op.OpLoader - Adding the traversal OpProcessor.