Multiple threads access to gremlin server with one client and authenticate failed

[Yong Fang]

Hi all

    I create gremlin server with authentication and I use org.apache.tinkerpop.gremlin.driver.Client to submit my query. I found the description in Client class shows that The Client is designed to be re-used and shared across threads, it means the methods in Client should be thread safe. Bu I found when I use the same instance of Client in multiple threads, it comes to some error as follows:

java.util.concurrent.CompletionException: org.apache.tinkerpop.gremlin.driver.exception.ResponseException: Failed to authenticate

at java.util.concurrent.CompletableFuture.reportJoin(CompletableFuture.java:375)

at java.util.concurrent.CompletableFuture.join(CompletableFuture.java:1934)

at org.apache.tinkerpop.gremlin.driver.ResultSet.one(ResultSet.java:94)

at org.apache.tinkerpop.gremlin.driver.ResultSet$1.hasNext(ResultSet.java:146)

at ****(ClientMain.java:34)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

Caused by: org.apache.tinkerpop.gremlin.driver.exception.ResponseException: Failed to authenticate

at org.apache.tinkerpop.gremlin.driver.Handler$GremlinResponseHandler.channelRead0(Handler.java:228)

at org.apache.tinkerpop.gremlin.driver.Handler$GremlinResponseHandler.channelRead0(Handler.java:184)

at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)

at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)

at org.apache.tinkerpop.gremlin.driver.Handler$GremlinSaslAuthenticationHandler.channelRead0(Handler.java:112)

at org.apache.tinkerpop.gremlin.driver.Handler$GremlinSaslAuthenticationHandler.channelRead0(Handler.java:66)

at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)

at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)

at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)

at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)

at org.apache.tinkerpop.gremlin.driver.handler.WebSocketClientHandler.channelRead0(WebSocketClientHandler.java:91)

at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)

at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)

at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)

at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)

at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)

at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)

at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)

at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)

at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:646)

at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:581)

at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)

at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:460)

at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)

... 1 more

    I debug in my local env and found that the gremlin server check the auth as follows:

1. client submit a query to server

2. server receive the query, and check the negotiator in ChannelHandlerContext ctx. if it's null, create negotiator and set it to ctx, write authenticate message to client

3. client receive the authenticate message and build authenticate request

4. server receive authenticate request, check the authentication and remove the authenticate from pipeline of netty, then get the original request from the authenticate message and do the query

    Suppose there're two thread as A and B, when A access gremlin server and reach to step3, and B has submit it's query and the gremlin server reach to step 2, the query from B will be "Failed to authenticate" when gremlin server can get negotiator instance from ctx which is set by A. Am I right? or could any give me some input about the Client for multiple threads? Thanks

[Stephen Mallette]

I believe that this is a known issue:

https://issues.apache.org/jira/browse/TINKERPOP-2132

There is a fix on a branch of that same name if you'd like to test it. I've not had a chance to fully test the work but I think it's solved there.

On Thu, Jan 31, 2019 at 3:49 AM Yong Fang <zju...@gmail.com> wrote:

By the way, my version of gremlin server is 3.2.3, thanks :)

在 2019年1月31日星期四 UTC+8下午4:46:15，Yong Fang写道：

--
You received this message because you are subscribed to the Google Groups "Gremlin-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gremlin-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/87a960aa-5128-4e91-97bb-2a4fa0cd2fbf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Yong Fang]

Glad to see the issue, and I hope this fix can be released as soon as possible, thanks again ^_^

在 2019年1月31日星期四 UTC+8下午7:30:59，Stephen Mallette写道：