Can't add a graph when the number of nodes are high - ERROR Handler$GremlinResponseHandler

[Dilan Ranasinghe]

Hello


I'm using janusgraph/gremlin server  and connect to it via gremlin driver 3.2.3.

The gremlin server and the application which connect to it runs on the same machine.

I need to create a graph with thousands of nodes and edges  (More than hundred thousand).

What i did is create a gremlin query at once for the creation of the whole graph and send to the server.

As the graph grows larger, the query i create for generating the graph is also getting larger and it seems the gremlin server fails to process the request.
I got the following error.

2017-09-26 07:26:54 ERROR Handler$GremlinResponseHandler:246 - Could not process the response
java.io.IOException: Connection reset by peer
        at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
        at sun.nio.ch.IOUtil.read(IOUtil.java:192)
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
        at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:221)
        at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:899)
        at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:241)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:572)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:513)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:427)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:399)
        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140)
        at java.lang.Thread.run(Thread.java:745)

So i increase the content length in the gremlin-server.yaml. After that i'm getting the following error in creating large graphs.

java.lang.RuntimeException: Method code too large!
        at groovyjarjarasm.asm.MethodWriter.a(Unknown Source)
        at groovyjarjarasm.asm.ClassWriter.toByteArray(Unknown Source)
        at org.codehaus.groovy.control.CompilationUnit$16.call(CompilationUnit.java:815)
        at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1053)
        at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:591)
        at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:569)
        at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:546)
        at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
        at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
        at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:254)
        at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:211)
        at org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine.getScriptClass(GremlinGroovyScriptEngine.java:527)
        at org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine.eval(GremlinGroovyScriptEngine.java:446)
        at javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:233)
        at org.apache.tinkerpop.gremlin.groovy.engine.ScriptEngines.eval(ScriptEngines.java:119)
        at org.apache.tinkerpop.gremlin.groovy.engine.GremlinExecutor.lambda$eval$2(GremlinExecutor.java:287)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

1 error

        at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
        at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
        at tsn.poc.GraphConnector.submitQuery(GraphConnector.java:40)
        at tsn.poc.OptimizedGraphBuilder.buildGraph(OptimizedGraphBuilder.java:157)
        at tsn.poc.Poc.main(Poc.java:44)
Caused by: org.apache.tinkerpop.gremlin.driver.exception.ResponseException: startup failed:
General error during class generation: Method code too large!



I need to know
what is the recommended way to add a large graph?
Do i need to add the graph part by part?
Or is there any configuration in the driver or the server to resolve the above issue?

Thanks
Dilan

[Stephen Mallette]

I sense that you are concatenating together a massive Gremlin script (many lines long) and shipping it to Gremlin Server for execution. That is an anti-pattern. It is far more efficient to send smaller parameterized requests to the server. So rather than send:

g.addV('person').property('name','alice').iterate()|

g.addV('person').property('name','bill').iterate()|

...

g.addV('person').property('name','zed').iterate()|

it would be much better to do:

for (p in persons) {

  g.addV('person').property('name',p.name).iterate()|

}

where "persons" is a parameter that contains a list of map objects. You can read more about parameterization here:

http://tinkerpop.apache.org/docs/current/reference/#parameterized-scripts

of course, assuming this is a one-time data load you may have even better results if you just write a groovy script and execute it in the Gremlin Console. That might be the easiest/fastest way to load data numbering in the "hundreds of thousands" of elements:

http://tinkerpop.apache.org/docs/current/tutorials/getting-started/#_loading_data

--

You received this message because you are subscribed to the Google Groups "Gremlin-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gremlin-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/c111f738-ec85-44d7-8cc1-cc7a4599dbd9%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Dilan Ranasinghe]

My requirement is to store 40000 transactions per minute which will cause the creation of 40000 nodes. This is not a one time process since i will get this much of updates constantly.
Even I try to create parameterized  scripts i may need to still pass the properties, labels etc to the gremlin server via the driver which seems throwing exceptions when the size of the request is too big.
So as i understood from the information available, even though janusgraph can cater for search requests with huge amount of data sets, it can't be used for real time updates as what i need.
Is my understanding correct?

To unsubscribe from this group and stop receiving emails from it, send an email to gremlin-user...@googlegroups.com.

[Stephen Mallette]

> Even I try to create parameterized  scripts i may need to still pass the properties, labels etc to the gremlin server via the driver which seems throwing exceptions when the size of the request is too big.

I don't see how you can end up with a "method too large" exception if you are parameterizing. "method too large" means you are exceeding the maximum size the JVM will accept for a method:

https://stackoverflow.com/q/3192896/1831717

Perhaps you should provide a sample of this script, so that we can get an idea as to how you are approaching things.

> So as i understood from the information available, even though janusgraph can cater for search requests with huge amount of data sets, it can't be used for real time updates as what i need. Is my understanding correct?

No. That conclusion is not correct. Your ability to sustain a high transaction rate however is going to be dictated by more than just sending a massive script to Gremlin Server. The size/power/configuration of the JanusGraph cluster will matter. The amount of paralellism you can incorporate into the load will matter. There are a number of factors involved which are probably better discussed on the JanusGraph mailing list.

To unsubscribe from this group and stop receiving emails from it, send an email to gremlin-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/5239a736-5957-4449-a490-b999400b8218%40googlegroups.com.

[Dilan Ranasinghe]

Hi Stephen,

Sorry for making a confusion here.
I had missed one of the places of query generation.
After i started making all as parameterized queries all works fine.
Sorry again.

Thanks.