App Engine standard and compressed requests

[Samu Lahti]

We use App Engine standard with Java 8. We have a client that sends HTTP POST (post body as JSON) requests as gzip-compressed. In normal cases, everything works just fine. The frontend will uncompress the requests and the App Engine will see the uncompressed request.

But with some requests (large ones?) uncompressing is not done by the frontend. This is seen as the request header content-encoding=[gzip] is still in the request headers and content is binary when the request arrives in App Engine. Our application is not prepared to uncompress the request but relies on the frontend to uncompress it. We couldn't find much information about the rules of uncompressing requests. The page https://cloud.google.com/appengine/docs/standard/java/how-requests-are-handled#request_limits says only that the request limit is 32 megabytes. Our failing requests are far below that limit in both compressed and uncompressed.

• Is there a way to find out why some requests are not uncompressed?
• If it is the size of the request, can the limit be configured/increased?
• Is the best practice that the application must handle compressed requests also even frontend is taking care of the majority of the compressed requests?
• Is there documentation somewhere telling more about compressed request limits/handling in App Engine standard?

[Felipe Bergallo Corral]

My understanding is that that 32MB constraint is only if the responses are not on Cloud Storage or Legacy Blobstore. If you're requesting gzip-compressed responses, then the responses shouldn't be uncompressed, as far as I understand it - according to this, if the request doesn't specify compressed responses ( by having the header Accept-Encoding: gzip) it should return uncompressed responses. The likely reason for some responses being compressed and others not is due to caching if you've configured, if you've configured it (though I believe there is a predetermined configuration).

Though I'm not sure how to answer your third question, would you mind rephrasing it? I'm not sure what you mean

[Samu Lahti]

Sorry not to be clear enough. I am talking about inbound traffic here, so HTTP POST requests to the App Engine. Not HTTP responses from the App Engine. 
Our responses from App Engine are zipped just fine by the platform, no problems there.

The case here really is HTTP POSTs to the App Engine when the HTTP request POST body is compressed. Normal cases are that App Engine will get the request as already uncompressed but sometimes the request is coming as is thus still compression in place. I am suspecting that it has something to do with the request size. I am not sure though. 

[Felipe Bergallo Corral]

So, just to get this straight because I can be a little slow on the uptake sometimes:
All of the requests you are sending to App Engine are uncompressed, but occassionally App Engine will receive them as being compressed, is that correct?

[Samu Lahti]

Just the opposite. All of the requests we are sending to App Engine are compressed. Normal case is that our App Engine Java app will receive them as being uncompressed. But occasionally App Engine Java app will receive them as still being uncompressed.

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/nbzRV2WctMY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/ee3fd588-dfb1-4935-b33e-a8cfe7e26a2cn%40googlegroups.com.

[Felipe Bergallo Corral]

Hmmm... Judging by this document it seems that occasionally compression is expected. It states that "depending on the request headers and response Content-Type, the server may automatically compress the response body". Doesn't specify which, so I understand that App Engine Java may respond as needed unless you specify a response Content-Type header.

Though this is from your client to GAE, I don't see why this couldn't occur in the opposite direction. 

What are you using as frontend? Is there a reason you're handling de-compression there?

[Samu Lahti]

As said earlier, this has nothing to do with responses from AppEngine. There are not any extra players in front of AppEngine either. This is a very basic case, a client making a request to AppEngine running Java 8 application.

[Client] ->  [Compressed request] -> [Google AppEngine  *** -> [ AppEngine Java 8 Application ]]

I am really looking for the information or documentation about what happens in AppEngine request processing (***) before the request reaches the Java application running in the AppEngine. 
The request is normally uncompressed automatically. Sometimes it is not and I am looking for the root cause of what causes it NOT to be un-compressed.

[M M]

Could you check this SO link, specifically Nick Johnson's comment related to using a "sensible" user-agent in addition to content type? I suspect these are being explicitly defined in your POST request. In addition, are you logging the exceptions when these compressed packages arrives to GAE? Are you capturing the size of the load of these packages? It would help here if those compressed packages that make it through have certain file size or if it fails to uncompressed for any size and it is more random. For example, these packages usually have 1MB of size or less? You specified that the packages are not more that 32MB. Uncompressed, are they more than this limit (Maybe not relevant? I am not in capacity to reproduce this problem, but this would be a good detail to share here, if possible)

Cheers,

[Samu Lahti]

SO post discuss about the compressing the response, so not directly relevant here. Anyway our client is a browser, so user-agent is "sensible" already like in case with Chrome.

 user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36

btw; We are seeing success and failing requests with the same browser, so I do not suspect that the user-agent header has anything to with this issue as it is the same in both cases.

Content type is sent like this:

content-type=application/json;charset=utf-8

We do have logging in place but we have not done any tests what the exact limit is when it starts to fail. We have seen failing requests where sizes have been about 1.5Mb compressed / ~10Mb uncompressed. Majority of requests are much less than that and we haven't had any problems with those smaller requests. That is why we suspect that it really has something to do with the requests size but as 10Mb is far less 32Mb which we found from one document, we are not sure. And it was a bit of surprise that sometimes the requests is not uncompressed like in normal cases. So we weren't prepared for that.

This is easy to handle as when the request is uncompressed content-encoding=gzip header is still in place. So such request can be detected and uncompressed in the application. But if AppEngine should take care of it (should it?) why we need backup processing in the application? So really would want to understand what are the limits and what we should be prepared for.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/eaf7fb6a-20cf-413f-88bf-39e5ac55d91en%40googlegroups.com.

[Samu Lahti]

I think that I found out something. I have misinterpreted the uncompressed sizes.

 
AppEngine front end works like this:
If the request is sent as uncompressed and POST body size is over 32Mb, AppEngine responses HTTP 413 Request Entity Too Large
If  the request is sent as compressed POST where compressed size is less than 32Mb but uncompressed content is over 32Mb, AppEngine will give that request to AppEngine Java application as unprocessed.

This is easy to test with curl. Just create a test file of wanted size (testfile.json in the example), replace AppEngine URL in the example and run the command:

cat testfile.json | gzip | curl -i --data-binary @- -H "Content-Encoding: gzip" -X POST "https://your-appengine-url.appspot.com/..." -H "accept: application/json; charset=UTF-8" -H "Content-Type: application/json; charset=UTF-8"

This will compress testfile.json and it is POSTed to AppEngine as compressed