Max frame length of 65536 has been exceeded

[Frank Panetta]

Hi,

I'm running a remote gremlin query using Titan 1.0 and T3 and get the following error:

11:05:49 WARN  org.apache.tinkerpop.gremlin.driver.handler.WebSocketClientHandler  - Exception caught during WebSocket processing - closing connection

io.netty.handler.codec.CorruptedFrameException: Max frame length of 65536 has been exceeded.

at io.netty.handler.codec.http.websocketx.WebSocket08FrameDecoder.protocolViolation(WebSocket08FrameDecoder.java:434)

at io.netty.handler.codec.http.websocketx.WebSocket08FrameDecoder.decode(WebSocket08FrameDecoder.java:235)

at io.netty.handler.codec.ReplayingDecoder.callDecode(ReplayingDecoder.java:370)

at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:168)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)

at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)

at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846)

at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)

at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)

at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)

at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)

at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)

at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)

at java.lang.Thread.run(Thread.java:745)

11:05:49 ERROR org.apache.tinkerpop.gremlin.driver.Handler$GremlinResponseHandler  - Could not process the response - correct the problem and restart the driver.

io.netty.handler.codec.CorruptedFrameException: Max frame length of 65536 has been exceeded.

at io.netty.handler.codec.http.websocketx.WebSocket08FrameDecoder.protocolViolation(WebSocket08FrameDecoder.java:434)

at io.netty.handler.codec.http.websocketx.WebSocket08FrameDecoder.decode(WebSocket08FrameDecoder.java:235)

at io.netty.handler.codec.ReplayingDecoder.callDecode(ReplayingDecoder.java:370)

at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:168)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)

at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)

at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846)

at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)

at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)

at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)

at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)

at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)

at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)

at java.lang.Thread.run(Thread.java:745)

Max frame length of 65536 has been exceeded.

My config is:

host: localhost

port: 8182

threadPoolWorker: 1

gremlinPool: 8

scriptEvaluationTimeout: 300000

serializedResponseTimeout: 300000

channelizer: org.apache.tinkerpop.gremlin.server.channel.WebSocketChannelizer

graphs: {

  graph: conf/gremlin-server/titan-cassandra-es.properties}

plugins:

  - aurelius.titan

scriptEngines: {

  gremlin-groovy: {

    imports: [java.lang.Math],

    staticImports: [java.lang.Math.PI],

    scripts: [scripts/empty-sample.groovy]},

  nashorn: {

      imports: [java.lang.Math],

      staticImports: [java.lang.Math.PI]}}

serializers:

  - { className: org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV1d0, config: { useMapperFromGraph: graph }}

  - { className: org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV1d0, config: { serializeResultToString: true }}

  - { className: org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerGremlinV1d0, config: { useMapperFromGraph: graph }}

  - { className: org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerV1d0, config: { useMapperFromGraph: graph }}

processors:

  - { className: org.apache.tinkerpop.gremlin.server.op.session.SessionOpProcessor, config: { sessionTimeout: 28800000 }}

metrics: {

  consoleReporter: {enabled: true, interval: 180000},

  csvReporter: {enabled: true, interval: 180000, fileName: /tmp/gremlin-server-metrics.csv},

  jmxReporter: {enabled: true},

  slf4jReporter: {enabled: true, interval: 180000},

  gangliaReporter: {enabled: false, interval: 180000, addressingMode: MULTICAST},

  graphiteReporter: {enabled: false, interval: 180000}}

threadPoolBoss: 1

maxInitialLineLength: 4096

maxHeaderSize: 8192

maxChunkSize: 8192

maxContentLength: 65536

maxAccumulationBufferComponents: 1024

resultIterationBatchSize: 64

writeBufferHighWaterMark: 32768

writeBufferHighWaterMark: 65536

ssl: {

  enabled: false}

and I'm running on Mac OS.

Thanks

Frank

[Stephen Mallette]

Hi Frank, welcome back to TinkerPop-land! I assume you are connecting with the java driver.  You should probably bump the size of the maxContentLength when you build your Cluster object.  Note that you should also set the maxContentLength configuration in your server YAML file to the same number you have on the driver side.

--
You received this message because you are subscribed to the Google Groups "Gremlin-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gremlin-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/17ce5704-d5dc-4488-87b7-9cc62674c49a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Frank Panetta]

Thank you Stephen, it's nice to be back. A lot has changed since I was here last. But its all good.

I was able to adjust maxContentLength in gremlin-server.yaml but couldn't figure out how to add it to my remote.yaml. When I added maxContentLength, it would error.

Tried searching the docs but couldn't find anything on it. Thanks

[Stephen Mallette]

It's on the connectionPool config:

hosts: [localhost]

port: 8182

connectionPool: {

  enableSsl: true

  maxContentLength: 1234567 }

serializer: { className: org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV1d0, config: { serializeResultToString: true }}

are you submitting something "big"? I thought that the default size was pretty good for most applications....

To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/ca0e5df1-bdc4-45db-8096-821837f5b992%40googlegroups.com.

[Frank Panetta]

I didn't think that it was too large. I'm doing a groupCount() and/or groupCount().by('name') on 2748 vertices, but I keep getting errors. The error I received above occurred when I removed serializer: { className: org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV1d0, config: { serializeResultToString: true }} from my remote.yaml. Adding it in gives me the following error:

01:39:47 WARN  org.apache.tinkerpop.gremlin.driver.MessageSerializer  - Response [SimpleLeakAwareByteBuf(PooledUnsafeDirectByteBuf(ridx: 1461, widx: 1461, cap: 1461))] could not be deserialized by org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV1d0.

01:39:47 ERROR org.apache.tinkerpop.gremlin.driver.Handler$GremlinResponseHandler  - Could not process the response - correct the problem and restart the driver.

io.netty.handler.codec.DecoderException: org.apache.tinkerpop.gremlin.driver.ser.SerializationException: java.lang.IndexOutOfBoundsException: Index: 47, Size: 0

at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:99)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)

at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)

at org.apache.tinkerpop.gremlin.driver.handler.WebSocketClientHandler.channelRead0(WebSocketClientHandler.java:90)

at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)

at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)

at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)

at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)

at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:182)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)

at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)

at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846)

at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)

at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)

at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)

at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)

at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)

at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)

at java.lang.Thread.run(Thread.java:745)

Caused by: org.apache.tinkerpop.gremlin.driver.ser.SerializationException: java.lang.IndexOutOfBoundsException: Index: 47, Size: 0

at org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV1d0.deserializeResponse(GryoMessageSerializerV1d0.java:216)

at org.apache.tinkerpop.gremlin.driver.handler.WebSocketGremlinResponseDecoder.decode(WebSocketGremlinResponseDecoder.java:49)

at org.apache.tinkerpop.gremlin.driver.handler.WebSocketGremlinResponseDecoder.decode(WebSocketGremlinResponseDecoder.java:36)

at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89)

... 20 more

Caused by: java.lang.IndexOutOfBoundsException: Index: 47, Size: 0

at java.util.ArrayList.rangeCheck(ArrayList.java:653)

at java.util.ArrayList.get(ArrayList.java:429)

at org.apache.tinkerpop.shaded.kryo.util.MapReferenceResolver.getReadObject(MapReferenceResolver.java:60)

at org.apache.tinkerpop.shaded.kryo.Kryo.readReferenceOrNull(Kryo.java:831)

at org.apache.tinkerpop.shaded.kryo.Kryo.readObjectOrNull(Kryo.java:725)

at org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV1d0.deserializeResponse(GryoMessageSerializerV1d0.java:199)

... 23 more

Index: 47, Size: 0

I applied the changes to maxContentLength (thank's for the config) in both gremlin-server.yaml and remote.yaml but still getting the error above. 

I should probably mention that I'm using TinkerPop version 3.0.1-incubating.

Thanks

[Stephen Mallette]

You must specify a serializer and it must match one that you have configured on the server.  

Do other queries work with the maxContentLength set? What happens with just simple stuff like:

1+1

g.V().next()

is that ok? I think you need to get back to a basic working state and then we can try to fix from there.

To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/0cee54ab-dd9e-4112-81e8-7b88fc5eb177%40googlegroups.com.

[Frank Panetta]

Hi Stephen,

Yes, I have gone back to a basic config and have specified the serializer and maxContentLength in both client and server drivers and made sure that they match. But unfortunately I am still getting errors

11:36:37 WARN  org.apache.tinkerpop.gremlin.driver.MessageSerializer  - Response [PooledUnsafeDirectByteBuf(ridx: 1994, widx: 1994, cap: 1994)] could not be deserialized by org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV1d0.

11:36:37 ERROR org.apache.tinkerpop.gremlin.driver.Handler$GremlinResponseHandler  - Could not process the response - correct the problem and restart the driver.

io.netty.handler.codec.DecoderException: org.apache.tinkerpop.gremlin.driver.ser.SerializationException: java.lang.IndexOutOfBoundsException: Index: 52, Size: 0

        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:99)

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)

        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)

        at org.apache.tinkerpop.gremlin.driver.handler.WebSocketClientHandler.channelRead0(WebSocketClientHandler.java:90)

        at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)

        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)

        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)

        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)

        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:182)

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)

        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)

        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846)

        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)

        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)

        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)

        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)

        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)

        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)

        at java.lang.Thread.run(Thread.java:745)

Caused by: org.apache.tinkerpop.gremlin.driver.ser.SerializationException: java.lang.IndexOutOfBoundsException: Index: 52, Size: 0

        at org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV1d0.deserializeResponse(GryoMessageSerializerV1d0.java:216)

        at org.apache.tinkerpop.gremlin.driver.handler.WebSocketGremlinResponseDecoder.decode(WebSocketGremlinResponseDecoder.java:49)

        at org.apache.tinkerpop.gremlin.driver.handler.WebSocketGremlinResponseDecoder.decode(WebSocketGremlinResponseDecoder.java:36)

        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89)

        ... 20 more

Caused by: java.lang.IndexOutOfBoundsException: Index: 52, Size: 0

        at java.util.ArrayList.rangeCheck(ArrayList.java:653)

        at java.util.ArrayList.get(ArrayList.java:429)

        at org.apache.tinkerpop.shaded.kryo.util.MapReferenceResolver.getReadObject(MapReferenceResolver.java:60)

        at org.apache.tinkerpop.shaded.kryo.Kryo.readReferenceOrNull(Kryo.java:831)

        at org.apache.tinkerpop.shaded.kryo.Kryo.readObjectOrNull(Kryo.java:725)

        at org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV1d0.deserializeResponse(GryoMessageSerializerV1d0.java:199)

        ... 23 more

Index: 52, Size: 0

I'm able to run the exact same query that generates the error message in the gremlin console from ipython notebook over a websocket and it runs and returns results (2600 rows) without issue. Not sure what's going on but I think I'm going to upgrade to v 3.0.2-incubating and see if that resolves the issue.

Thanks

[Stephen Mallette]

The script of 1+1 should work on any version. If that's not working, I'm not sure what's going on.  Maybe drop Titan, download Gremlin Server and just try some basic operations with just TinkerGraph.  I think you have to find some kind of working state in order to debug the original problem.

To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/d4da1bd6-8331-4aab-a819-f61dda87e07f%40googlegroups.com.

[Frank Panetta]

Oh, no 1+1 is definitely working. Sorry I wasn't clear about that. No, I'm able to run basic Gremlin queries against the Titan database but once they get a bit more complicated it starts to get a little flakey.

The query that is giving me the most trouble includes a groupCount. If I run it by limiting the range(0, 255) I get results. If I increase the range by 1 i.e. range(0, 256) it errors. Then once the server errors it gets itself into a state where a normal query won't run on the first request but will run on the subsequent request then won't run on the subsequent request and so forth. The only way to resolve this is by reinitialising the connection to the server by using a remote: command.

So, in short, it all works until it trys to process a query that requires a little more processing.

...

[Frank Panetta]

But the exact same queries work fine when running from ipython. 

[Stephen Mallette]

I don't know the error to manifest as a content length problem, but the weirdness with limit() sounds familiar to me as a bug with 3.0.x.  I believe that gets resolved for 3.1.x. You might want to try to build the titan11 branch with: 

mvn clean install -DskipTests=true -Paurelius-release -Dgpg.skip=true

The zip distributions get built to titan-dist/target.  Sorry for the hassle there....

--

You received this message because you are subscribed to the Google Groups "Gremlin-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gremlin-user...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/d18fbd06-45bf-4332-99e0-a0454f1b5ebd%40googlegroups.com.

[Frank Panetta]

No problem. I'll upgrade and see how it goes.

Thanks

You received this message because you are subscribed to a topic in the Google Groups "Gremlin-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/gremlin-users/JWto9uv8CeA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to gremlin-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/CAA-H43_SiC6NeUFJxG_e9KvL7tdZaAzsozUkrOXw-Jhvc5vDTw%40mail.gmail.com.