Max frame length of 65536 has been exceeded error seen intermittently.

[Anya Sharma]

I have a set up where I am using the gremlin-core java library to connect to a remote janusgraph server. The size of the database is moderate for now but will increase in the future. I saw the Max frame length of 65536 has been exceeded error before and realized that I am sending a large number of vertex ids as an array to fetch the vertices in one of my queries. I applied a batch to the code and fixed the issue. 

I then saw this error again for a simple g.V() query. As I said, the database is not that large and I could not figure out why this error would show up even though I am not sending any parameters to the janusgraph server in my query. But the query failed with this error if the result size was more than 63 vertices.

In one of the replies to the same issue in this group, I came across a proposed solution which mentioned that I need to add the maxContentLength parameter on the client yaml file as well. I did that and the issue was resolved. But it brings up a few questions about the configuration parameters and their values with respect to the size of the database.

Q.1 Is there a link between the size of the response of a query to the maxContentLength parameter? Is there any other parameter which defines the maximum size of the response?

Q.2 What is the ratio/proportion of the configuration parameters to the size of the database? In other words, how do I determine the safe value of a parameter with respect to the size of the database so as to avoid this error in the future?

Thanks in advance

Anya Sharma

[Stephen Mallette]

maxContentLength is the number of bytes a single "message" can contain as a request or a response. It serves the same function as similar settings in web servers to allow filtering of obviously invalid requests. The setting has little to do with database size and more to do with the types of requests you are making and the nature of your results. For requests, I tend to think it atypical for a request to exceed 65k in most situations. Folks who exceed that size are typically trying to do batch loading or are using code generated scripts (the latter is typically problematic, but I won't go into details). For responses, 65k may not be enough depending on the nature of your queries. For example, the query:

g.V().valueMap(true)

will return all vertices in your database as an Iterator<Map> and Gremlin Server will stream those result back in batches controlled by the resultIterationBatchSize (default is 64). So if you have 128 vertices in your database Gremlin Server will stream back two batches of results behind the scenes. If those two batches are each below maxContentLength in size then no problems. If your batches are bigger than that because you have 1000 properties on each vertex then you either need to 

1. limit the data you return - e.g.  return fewer properties

2. increase maxContentLength

3. lower the resultIterationBatchSize

Also note that the previous query is very different from something like:

g.V().valueMap(true).fold()

because the fold() will realize all the vertices into a list in memory and then that list must be serialized as a whole. There is only 1 result (i.e. List<Map> with 128 vertices) and thus nothing to batch, so its much more likely that you would exceed the maxContentLength here and lowering the resultIterationBatchSize wouldn't even help. You're only recourse would be to increase maxContentLength or alter the query to allow batching to kick in to hopefully break up that large chunk of data to fit in the maxContentLength.

Setting your maxContentLength to 2mb or larger shouldn't be too big a deal. If you need to go higher for requests, then i'd be curious what the reason was for that. If you need to go much higher for responses, then perhaps I'd take a look at my queries and see if there's a better way to limit the data i'm returning or to see if there's a nicer way to get Gremlin Server streaming to work for me.

--
You received this message because you are subscribed to the Google Groups "Gremlin-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gremlin-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/741c1d44-aef8-4581-9a74-2b6341b05762%40googlegroups.com.

[Anya Sharma]

Hi Stephen,

Thanks for the response. The queries that I fire are run-time queries that I process in the code using gremlin-core libraries. (Specifically, the functions Client.submit() for sending the query to the server and ResultSet.all() to process the response). Since the nature of the query is unknown to me before I send it to the server, my fault tolerance options are limited. 

The valueMap function will not work for me as I use Edge and Vertex functions on the response like id(), inVertex(), outVertex() etc in my code. The valueMap function returns a Map that will not support any of these functions.

I did a load test on the two parameters that you mentioned - maxContentLength and resultIterationBatchSize. What I found was that no matter what the value of the resultIterationBatchSize parameter is, as long as the value of maxContentLength parameter is low, I will get the above error. I am testing with g.V() and g.E() queries as these return the responses with the highest load. So for around 1000 edges and 400 vertices:

maxContentLength = 65536, resultIterationBatchSize = 64/32/16 - give max frame length exceeded error.

maxContentLength = 655369, resultIterationBatchSize = 64/32/16 - works.

As you mentioned, increasing the maxContentLength value to 2MB is not a big deal, my concern is how do I be sure that it is enough. Specifically, how do I determine the cut off for the number of edges or vertices that I return so that I do not get this error?

Just to be sure that the configuration that I am using is correct, I am specifying maxContentLength and resultIterationBatchSize in 2 places - server-side yaml file and on the client-side yaml file with the same values. Is this correct? 

On the client-side, I am using ResultSet.all() method to process the response. Is there a way that I can apply batching in the client-side code explicitly to avoid this error? 

Anya

[Stephen Mallette]

Comments inline below:

 

Thanks for the response. The queries that I fire are run-time queries that I process in the code using gremlin-core libraries. (Specifically, the functions Client.submit() for sending the query to the server and ResultSet.all() to process the response). Since the nature of the query is unknown to me before I send it to the server, my fault tolerance options are limited. 

It seems to me that you should be able to make some basic assumption about how your application is expected to work. At some point the size of the data returned in a single message for a single query would have to reach an absurd level for your application. Maybe someone could write a query that returns gigabytes over gigabytes of data (for a single 64 item result iteration)....but would you want that? I mean....I suppose that if none of this means anything to you, just set the value to something so arbitrarily high it can't logically be exceeded.

 

The valueMap function will not work for me as I use Edge and Vertex functions on the response like id(), inVertex(), outVertex() etc in my code. The valueMap function returns a Map that will not support any of these functions.

That was just an example to explain how you can overpack a single 64 item result batch. Speaking specifically to your response here though, you can and should return your data as Map. If that means using project() to get the inV() and outV() for an edge, then you should do that. Note that 3.4.3 will come with elementMap() which will make all of that more simple:

https://github.com/apache/tinkerpop/pull/1189/files#diff-32f91d4edfd5fffafe8aa34fac03c87dR32-R101  

 

I did a load test on the two parameters that you mentioned - maxContentLength and resultIterationBatchSize. What I found was that no matter what the value of the resultIterationBatchSize parameter is, as long as the value of maxContentLength parameter is low, I will get the above error. I am testing with g.V() and g.E() queries as these return the responses with the highest load. So for around 1000 edges and 400 vertices:

maxContentLength = 65536, resultIterationBatchSize = 64/32/16 - give max frame length exceeded error.

maxContentLength = 655369, resultIterationBatchSize = 64/32/16 - works.

I mean....64k is kinda small so perhaps you are returning a lot of data and 16 still isn't small enough to get under that limit. If you're returning entire graph elements with all their properties and those properties are thick with big strings you're going to exceed that 64k pretty fast. I mean, imagine a vertex that has 1 million multi-properties on it........

This is why we recommend that you transform your graph elements to Map and only include the data that you need for your application to work. It's no different than SELECT * FROM table in SQL - you wouldn't do that, right? You would specify each of the individual fields that you wanted returned in the SELECT portion of the query. 

 

As you mentioned, increasing the maxContentLength value to 2MB is not a big deal, my concern is how do I be sure that it is enough. Specifically, how do I determine the cut off for the number of edges or vertices that I return so that I do not get this error?

As I tried to allude to in my previous response, it's not the "number of edges or vertices" that is at issue. The problem is with how many bytes of data you are returning from your query. Only you can determine that by knowing the nature of your data and understand the types of queries that you are sending to the server. Again, is it reasonable for someone to write this query:

g.V().fold()

That's going to return a ton of bytes because in a single result batch you will have every vertex in the graph in a List. To me that's not reasonable. I'd want to get an error for that. I'd want to then note that there was a traversal trying to return a ridiculous amount of data that was likely going to tie up resources on the server. That would then tell me that I should instead rewrite that traversal to:

g.V() 

to ensure that I stream my results back in smaller batches. Still not a great query, of course, as it's returning all vertices in my graph, but I'm just trying to demonstrate the thinking. As for the 2MB suggestion, I just threw that out there as an idea. Only you know what can possibly make sense in your environment. It might help to think about how much memory are you willing to lock up on the server to realize a result set for each message returned from the server. 

Just to be sure that the configuration that I am using is correct, I am specifying maxContentLength and resultIterationBatchSize in 2 places - server-side yaml file and on the client-side yaml file with the same values. Is this correct? 

maxContentLength should be set on server and client and typically should be the same value. resultIterationBatchSize can be set globally on the server then overridden by whatever setting you provide from the client side (I think that's a per request option).

 

On the client-side, I am using ResultSet.all() method to process the response. Is there a way that I can apply batching in the client-side code explicitly to avoid this error? 

Batching (i.e. streaming) occurs behind the scenes - you can't really control it directly. As I mentioned earlier however, you can write queries that allow streaming to kick in by ensuring that Gremlin results are shaped in a fashion to take advantage of that functionality.

I hope that helps clarify what you need to do. I think you just need to choose a sane value for the maxContentLength based on your environment and once you determine that sane value, examine queries that force the error to determine why they exceed your assumptions. At that point you either increase the maxContentLength again or you fix your query (preferably the latter).

To unsubscribe from this group and stop receiving emails from it, send an email to gremli...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/741c1d44-aef8-4581-9a74-2b6341b05762%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Gremlin-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gremlin-user...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/9318c00d-1299-4996-9461-8457be9c81c1%40googlegroups.com.