upstream max stream duration reached

[Bryan Gillespie]

Under what circumstances is "upstream max stream duration reached" sent back to the client?

I have a Google Cloud Platform application running in Cloud Run. The client is .NET HTTP/1.1, and the server is h2c (HTTP2 cleartext) with a Google Load Balancer in between. As you may know, Google Cloud Load Balancers are powered by Envoy.

Every once in a while (maybe a few times a day), a file upload (HTTP POST) to Cloud Run will fail and the response the client gets is "upstream max stream duration reached". From the server's perspective, I see a stream reset ("http2: server read frame RST_STREAM").

I'm having a hard time troubleshooting this. I appears (but have not 100% confirmed because the problem is intermittent) that turning off HTTP2 in Cloud Run "fixes" the issue. However, if I do that, I can no longer upload larger than 32MB since google front ends limit HTTP/1 requests to <32MB.

Anyone have any insights as to what could be happening here?

Thanks,

Bryan

[Yan Avlasov]

There are a few load balancers between a client and a cloud run application. Not all of them may be Envoy's.

From your description it looks like an intervening load balancer hits max request duration. Can you determine from your app logs the actual time between the start and end of such requests? You can see if adjusting request timeout on cloud run helps this issue or not.

--
You received this message because you are subscribed to the Google Groups "envoy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to envoy-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/envoy-users/c6aed575-575e-4cd7-bf32-93d4ccf4f791n%40googlegroups.com.