gremlin-driver where to set the heartbeat? getting java.nio.channels.ClosedChannelException

[Venkata Phani Kumar Mangipudi]

I've a microservice built in scala talking to gremlin-server (single host) via gremlin-driver. I am using titan11 branch. The gremlin-server is behind a nginx proxy. 

My service is running fine immediately after launch. But after say 1 minute, the service starts to fail with 

java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.nio.channels.ClosedChannelException
ms-app-consumer |       at org.apache.tinkerpop.gremlin.driver.Client.submit(Client.java:146)
ms-app-consumer |       at in.nixinn.services.Service$class.fetchConsumer(ConsumerMicroService.scala:660)
ms-app-consumer |       at in.nixinn.services.ConsumerMicroService$.fetchConsumer(ConsumerMicroService.scala:710)
....
....
java.nio.channels.ClosedChannelException: null
ms-app-consumer |       at akka.stream.impl.fusing.GraphInterpreterShell.runBatch(ActorGraphInterpreter.scala:44
1)
ms-app-consumer |       at akka.stream.impl.fusing.GraphInterpreterShell.receive(ActorGraphInterpreter.scala:396
....
....
ms-app-consumer | Caused by: java.util.concurrent.ExecutionException: java.nio.channels.ClosedChannelException
ms-app-consumer |       at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
ms-app-consumer |       at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
ms-app-consumer |       at org.apache.tinkerpop.gremlin.driver.Client.submit(Client.java:144)
ms-app-consumer |       ... 65 more
ms-app-consumer | Caused by: java.nio.channels.ClosedChannelException

When I increased the proxy_read_timeout on nginx, the issue is not occurring. Hence, I believe the websocket channel is being closed by nginx (nginx has a default value for the proxy_read_timeout as 60 seconds, which I now changed for the time being to 86400s). I was trying to figure out how and where I can configure heartbeat for the connections from gremlin-driver. Kindly help me with this.

Thanks and Regards,

Phani

[Stephen Mallette]

I had thought that the netty client was sending a ping frame automatically, but i don't see a configuration for it nor do i see in the code where it is doing anything other than handling it when it comes in from the server.  I think the driver would need to be updated to periodically issue the ping. It would be great if you could create an issue for this in jira so that we can address it in a future release.

--
You received this message because you are subscribed to the Google Groups "Gremlin-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gremlin-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/f134484a-42c5-4e18-bbf8-d54538c67687%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Venkata Phani Kumar Mangipudi]

Thank you Stephen for the inputs. Could you think off any workaround until the functionality is implemented? Having a large value for  proxy_read_timeout is really not a good idea. I explained in detail in the JIRA ticket:  https://issues.apache.org/jira/browse/TINKERPOP-1249

I couldn't think off any other possibility other than having the proxy_read_timeout!

Thanks and Regards,

Phani

[Stephen Mallette]

I'm still a bit surprised that netty wasn't handling this, but I haven't looked at it in detail yet. Perhaps you could start by confirming that it doesn't.  Then, if it doesn't, I guess you'd have to implement your own heartbeat of some kind to keep that connection through the proxy alive. Maybe send a "1+1" script as a heartbeat?

To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/ac4a7c5d-e48f-4e7d-b6d5-1afe205e7e31%40googlegroups.com.

[kARTHIK R]

I couldn't find the JIRA on this. Is this change rolled out to the driver?

[Stephen Mallette]

The JIRA is still 

https://issues.apache.org/jira/browse/TINKERPOP-1249

and no - it is not yet completed. I did give it a shot at one point thinking it was really simple, but then realized it was more troubles than i thought. I actually was hoping to try again next week as we prepare for 3.2.2/3.1.4 release coming up soon.

To unsubscribe from this group and stop receiving emails from it, send an email to gremlin-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/49cbddf6-ce3d-4227-9e6d-b3f93b16ae64%40googlegroups.com.