Envoy not propagating GOAWAY back to downstream client from upstream server.

Ajat Prabha

unread,

Jul 12, 2022, 1:47:46 AM7/12/22

to envoy-users

Hi team,

I’m encountering a problem with the Istio based envoy ingress gateway that I have used as the entry point to services in my cluster.

The issue is that this ingress is using HPA to scale up/down pods, and the gRPC connections created during the minimum pod count duration are not rebalanced to other pods when the pod count is higher after scale up.

Because of this, few envoy pods end up serving most of the gRPC requests and P99 latencies on client is impacted negatively during peak throughout. The gRPC server has a max_connection_age of 5 minutes, after which it will send a GOAWAY and in my understanding, envoy is not propagating this to gRPC client and it is still maintaining the connection to envoy.

Is there a way to solve this rebalancing issue for gRPC connections when auto-scale events happen? Maybe some configuration that can help.

PS: Attaching some metric screenshots as well.

Screenshot 2022-07-08 at 2.38.12 PM.png

envoy.png

Screenshot 2022-07-06 at 6.36.58 PM.png

Screenshot 2022-07-08 at 2.38.20 PM.png

Yan Avlasov

unread,

Jul 12, 2022, 9:52:53 AM7/12/22

to Ajat Prabha, envoy-users

You also need to configure https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/core/v3/protocol.proto#envoy-v3-api-field-config-core-v3-httpprotocoloptions-max-connection-duration on Envoy.

--
You received this message because you are subscribed to the Google Groups "envoy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to envoy-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/envoy-users/15a2f3ec-5bf5-45c2-b499-6eb508c9d55an%40googlegroups.com.

Ajat Prabha

unread,

Jul 13, 2022, 3:21:04 AM7/13/22

to envoy-users

Hi, thanks for pointing me to the correct documentation.

I believe max_connection_duration will work for my use-case, however, I can't seem to find relevant configuration option for max_connection_duration in Istio (I'm using it as the control plane) docs at https://istio.io/latest/docs/.

Can this property be configured via an EnvoyFilter or changes in Istio will be required to achieve this? Or some other way of telling Istio to use this configuration?

TIA

Matt Klein

unread,

Jul 13, 2022, 11:22:06 AM7/13/22

to Ajat Prabha, envoy-users

> Or some other way of telling Istio to use this configuration?

Please route Istio questions to Istio.

Thanks,

Matt

To view this discussion on the web visit https://groups.google.com/d/msgid/envoy-users/5da8ae57-c1c1-4459-b946-07cc0b3214e9n%40googlegroups.com.

Reply all

Reply to author

Forward