GraphSON: Serialization fails for larger data sets.

324 views
Skip to first unread message

Charles Zilm

unread,
Jan 20, 2016, 12:34:52 PM1/20/16
to Gremlin-users
All,

I'm running TitanDB 1.0.0 with berkleyje as the storage backend. I've written a small application that acts as a driver for the gremlin server over web sockets. When sending a query that would return more than a handful of vertices (>50) I get the error trace that follows this message. 

Could this be a configuration issue? IF so what parameters should I look at?

Is there an upper limit on how many objects can be serialized in a response?

If there is what is the best practice for retrieving larger result sets? My first thought is to use sessions to preserve the state of the result set and send responses back via range until an empty result set is received. Would like to avoid this overhead if possible.

Thank you,

Charles

431209389 [gremlin-server-exec-6] WARN  com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx  - Query requires iterating over all vertices [()]. For better performance, use indexes

431209427 [gremlin-server-worker-1] WARN  org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerV1d0  - Response [ResponseMessage{requestId=2549d964-bf8e-11e5-b2a1-0242ac1100bb, status=ResponseStatus{code=PARTIAL_CONTENT, message='', attributes={}}, result=Re

sponseResult{data=[v[266256], v[270352], v[274448], v[278544], v[282640], v[286736], v[290832], v[294928], v[299024], v[303120], v[307216], v[311312], v[315408], v[319504], v[323600], v[327696], v[331792], v[335888], v[339984], v[344080], v[348176], v[352272], v[356368]

, v[360464], v[364560], v[368656], v[372752], v[376848], v[380944], v[385040], v[389136], v[393232], v[397328], v[401424], v[405520], v[409616], v[413712], v[417808], v[421904], v[426000], v[430096], v[434192], v[438288], v[442384], v[446480], v[450576], v[454672], v[45

8768], v[462864], v[466960], v[471056], v[475152], v[479248], v[483344], v[487440], v[491536], v[495632], v[499728], v[503824], v[507920], v[512016], v[516112], v[520208], v[524304]], meta={}}}] could not be serialized by org.apache.tinkerpop.gremlin.driver.ser.Abstract

GraphSONMessageSerializerV1d0.

431209428 [gremlin-server-worker-1] WARN  org.apache.tinkerpop.gremlin.server.handler.WsGremlinResponseEncoder  - The result [ResponseResult{data=[v[266256], v[270352], v[274448], v[278544], v[282640], v[286736], v[290832], v[294928], v[299024], v[303120], v[307216], v[

311312], v[315408], v[319504], v[323600], v[327696], v[331792], v[335888], v[339984], v[344080], v[348176], v[352272], v[356368], v[360464], v[364560], v[368656], v[372752], v[376848], v[380944], v[385040], v[389136], v[393232], v[397328], v[401424], v[405520], v[409616

], v[413712], v[417808], v[421904], v[426000], v[430096], v[434192], v[438288], v[442384], v[446480], v[450576], v[454672], v[458768], v[462864], v[466960], v[471056], v[475152], v[479248], v[483344], v[487440], v[491536], v[495632], v[499728], v[503824], v[507920], v[5

12016], v[516112], v[520208], v[524304]], meta={}}] in the request 2549d964-bf8e-11e5-b2a1-0242ac1100bb could not be serialized and returned.

org.apache.tinkerpop.gremlin.driver.ser.SerializationException: com.fasterxml.jackson.databind.JsonMappingException: Could not execute operation due to backend exception (through reference chain: java.util.ArrayList[0])

        at org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerV1d0.serializeResponseAsString(GraphSONMessageSerializerV1d0.java:101)

        at org.apache.tinkerpop.gremlin.server.handler.WsGremlinResponseEncoder.encode(WsGremlinResponseEncoder.java:79)

        at org.apache.tinkerpop.gremlin.server.handler.WsGremlinResponseEncoder.encode(WsGremlinResponseEncoder.java:45)

        at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:89)

        at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:633)

        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:691)

        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:626)

        at org.apache.tinkerpop.gremlin.server.handler.IteratorHandler.write(IteratorHandler.java:119)

        at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:633)

        at io.netty.channel.AbstractChannelHandlerContext.access$1900(AbstractChannelHandlerContext.java:32)

        at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.write(AbstractChannelHandlerContext.java:908)

        at io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.write(AbstractChannelHandlerContext.java:960)

        at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:893)

        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:380)

        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)

        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)

        at java.lang.Thread.run(Thread.java:745)

Caused by: com.fasterxml.jackson.databind.JsonMappingException: Could not execute operation due to backend exception (through reference chain: java.util.ArrayList[0])

        at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:232)

        at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:211)

        at com.fasterxml.jackson.databind.ser.std.StdSerializer.wrapAndThrow(StdSerializer.java:212)

        at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:105)

        at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:21)

        at com.fasterxml.jackson.databind.ser.std.AsArraySerializerBase.serialize(AsArraySerializerBase.java:183)

        at org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONUtil.writeWithType(GraphSONUtil.java:49)

        at org.apache.tinkerpop.gremlin.driver.ser.AbstractGraphSONMessageSerializerV1d0$ResponseMessageSerializer.ser(AbstractGraphSONMessageSerializerV1d0.java:235)

        at org.apache.tinkerpop.gremlin.driver.ser.AbstractGraphSONMessageSerializerV1d0$ResponseMessageSerializer.serialize(AbstractGraphSONMessageSerializerV1d0.java:205)

        at org.apache.tinkerpop.gremlin.driver.ser.AbstractGraphSONMessageSerializerV1d0$ResponseMessageSerializer.serialize(AbstractGraphSONMessageSerializerV1d0.java:197)

        at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:114)

        at com.fasterxml.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:2811)

        at com.fasterxml.jackson.databind.ObjectMapper.writeValueAsString(ObjectMapper.java:2268)

        at org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerV1d0.serializeResponseAsString(GraphSONMessageSerializerV1d0.java:98)

        ... 16 more

Caused by: com.thinkaurelius.titan.core.TitanException: Could not execute operation due to backend exception

        at com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:44)

        at com.thinkaurelius.titan.diskstorage.BackendTransaction.executeRead(BackendTransaction.java:428)

        at com.thinkaurelius.titan.diskstorage.BackendTransaction.edgeStoreQuery(BackendTransaction.java:253)

        at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.edgeQuery(StandardTitanGraph.java:413)

        at com.thinkaurelius.titan.graphdb.query.vertex.SimpleVertexQueryProcessor$2.lambda$get$45(SimpleVertexQueryProcessor.java:114)

        at com.thinkaurelius.titan.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:84)

        at com.thinkaurelius.titan.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:76)

        at com.thinkaurelius.titan.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:68)

        at com.thinkaurelius.titan.graphdb.query.vertex.SimpleVertexQueryProcessor$2.get(SimpleVertexQueryProcessor.java:114)

        at com.thinkaurelius.titan.graphdb.query.vertex.SimpleVertexQueryProcessor$2.get(SimpleVertexQueryProcessor.java:111)

        at com.thinkaurelius.titan.graphdb.vertices.CacheVertex.loadRelations(CacheVertex.java:53)

        at com.thinkaurelius.titan.graphdb.query.vertex.SimpleVertexQueryProcessor.getBasicIterator(SimpleVertexQueryProcessor.java:111)

        at com.thinkaurelius.titan.graphdb.query.vertex.SimpleVertexQueryProcessor.iterator(SimpleVertexQueryProcessor.java:68)

        at com.google.common.collect.Iterables$8.iterator(Iterables.java:713)

        at com.thinkaurelius.titan.graphdb.query.vertex.SimpleVertexQueryProcessor.vertexIds(SimpleVertexQueryProcessor.java:91)

        at com.thinkaurelius.titan.graphdb.query.vertex.BasicVertexCentricQueryBuilder.executeIndividualVertices(BasicVertexCentricQueryBuilder.java:314)

        at com.thinkaurelius.titan.graphdb.query.vertex.BasicVertexCentricQueryBuilder.executeVertices(BasicVertexCentricQueryBuilder.java:309)

        at com.thinkaurelius.titan.graphdb.query.vertex.BasicVertexCentricQueryBuilder$VertexConstructor.getResult(BasicVertexCentricQueryBuilder.java:225)

        at com.thinkaurelius.titan.graphdb.query.vertex.BasicVertexCentricQueryBuilder$VertexConstructor.getResult(BasicVertexCentricQueryBuilder.java:221)

        at com.thinkaurelius.titan.graphdb.query.vertex.VertexCentricQueryBuilder.execute(VertexCentricQueryBuilder.java:77)

        at com.thinkaurelius.titan.graphdb.query.vertex.VertexCentricQueryBuilder.vertices(VertexCentricQueryBuilder.java:105)

        at com.thinkaurelius.titan.graphdb.vertices.AbstractVertex.getVertexLabelInternal(AbstractVertex.java:108)

        at com.thinkaurelius.titan.graphdb.vertices.AbstractVertex.vertexLabel(AbstractVertex.java:113)

        at com.thinkaurelius.titan.graphdb.vertices.AbstractVertex.label(AbstractVertex.java:104)

        at org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONSerializers$VertexJacksonSerializer.ser(GraphSONSerializers.java:191)

        at org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONSerializers$VertexJacksonSerializer.serialize(GraphSONSerializers.java:175)

        at org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONSerializers$VertexJacksonSerializer.serialize(GraphSONSerializers.java:163)

        at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:100)

        ... 26 more

Caused by: com.thinkaurelius.titan.diskstorage.PermanentBackendException: Permanent failure in storage backend

        at com.thinkaurelius.titan.diskstorage.berkeleyje.BerkeleyJEKeyValueStore.getSlice(BerkeleyJEKeyValueStore.java:166)

        at com.thinkaurelius.titan.diskstorage.keycolumnvalue.keyvalue.OrderedKeyValueStoreAdapter.getSlice(OrderedKeyValueStoreAdapter.java:56)

        at com.thinkaurelius.titan.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:65)

        at com.thinkaurelius.titan.diskstorage.BackendTransaction$1.call(BackendTransaction.java:256)

        at com.thinkaurelius.titan.diskstorage.BackendTransaction$1.call(BackendTransaction.java:253)

        at com.thinkaurelius.titan.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:56)

        at com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:42)

        ... 53 more

Caused by: java.lang.IllegalStateException: Transaction Id 693321 has been closed.

        at com.sleepycat.je.Transaction.checkOpen(Transaction.java:858)

        at com.sleepycat.je.Cursor.checkTxnState(Cursor.java:4077)

        at com.sleepycat.je.Cursor.search(Cursor.java:2393)

        at com.sleepycat.je.Cursor.getSearchKeyRange(Cursor.java:1727)

        at com.thinkaurelius.titan.diskstorage.berkeleyje.BerkeleyJEKeyValueStore.getSlice(BerkeleyJEKeyValueStore.java:123)

        ... 59 more


Stephen Mallette

unread,
Jan 20, 2016, 12:48:33 PM1/20/16
to Gremlin-users
There shouldn't be any limits to the size of the result (short of available memory to handle the result set i suppose).  There were a number of improvements to Gremlin Server transaction management for the soon to be released 3.1.1-incubating.  I can't say I saw this particular problem with Titan, but I have this strong sense it is addressed for that version.  You might want to try to build Titan 1.1.0 and bump to the TinkerPop 3.1.1-SNAPSHOT:


you can build the zip distributions with:

mvn clean install -DskipTests=true -Paurelius-release -Dgpg.skip=true





--
You received this message because you are subscribed to the Google Groups "Gremlin-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gremlin-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/27951e56-66ac-47d6-8bf0-b43b3bd9be56%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Charles Zilm

unread,
Jan 25, 2016, 11:38:23 AM1/25/16
to Gremlin-users
Thanks, I'll try upgrading Titan and TinkerPop when I can.

While digging into this I noticed something that seems a tad off.

Graph.traversal().V().limit(64) - 200 results(1-64)
Graph.traversal().V().limit(65) - 599 Serialization Error, 200 results(65)
Graph.traversal().V().limit(129) - 599 Serialization Error, 206 results(65-128), 200 results(129)

For queries that return 64 or fewer elements the results serialize correctly. For queries returning > 64 results a 599 is thrown for the first batch and the remaining results are then streamed as expected. According to the documentation any non 206 response code is to be seen as terminating. Is it intended behavior to have a response continue after an error is reported?

Stephen Mallette

unread,
Jan 26, 2016, 8:09:02 AM1/26/16
to Gremlin-users
I'm not able to recreate that behavior. Are you using gremlin-driver? or another client? 

Charles Zilm

unread,
Jan 26, 2016, 10:55:21 AM1/26/16
to Gremlin-users
This is the behavior I see in the erlang "driver" I'm putting together. Which at it's core is nothing more than a glorified websocket request manager.

Starts by opening a websocket with HttpHeaders = [
{language, "gremlin-groovy"},
{session, "false"},
{op, "eval"},
{processor, ""},
{accept, "application/json"}
],.

Wrap a script in a JSON object.
JSON = #{
<<"requestId">> => UUID,
<<"processor">> => <<"">>,
<<"op">>        => <<"eval">>,
<<"args">>      => #{
<<"gremlin">>  =>  Script,
<<"bindings">> => #{},
<<"accept">>   => <<"application/json">>,
<<"language">> => <<"gremlin-groovy">>
}
}.

Encode and send across the wire. Results given earlier are the raw messages received back from TitanDB.

Stephen Mallette

unread,
Jan 26, 2016, 2:51:54 PM1/26/16
to Gremlin-users
very cool to hear that you're getting an erlang driver built.  please keep us updated on your work. we'd love to link to your project and promote it when you get to a releasable state. you might want to subscribe to the developer mailing list as we often discuss gremlin server/driver stuff there:

http://mail-archives.apache.org/mod_mbox/incubator-tinkerpop-dev/

as for the problem, i wonder if the java driver is hiding this behavior.  i'll was analyzing it that way and not at the raw protocol level.  i wonder if one of the other driver devs might have an easy way to test this out or not....please create an issue in JIRA so we can track it. That would be helpful:




To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/7e00d704-8d62-4266-a134-8294199d1f58%40googlegroups.com.

Stephen Mallette

unread,
Feb 2, 2016, 6:52:30 AM2/2/16
to Gremlin-users
Appended this problem to this issue:

Charles Zilm

unread,
Feb 11, 2016, 4:43:48 PM2/11/16
to Gremlin-users
Sorry I didn't get to reply to this sooner. Updating versions as suggested solved the serialization problems I was seeing.
Reply all
Reply to author
Forward
0 new messages