Request Retry

[Rama Rao]

Hi,

Is Request Retry enabled by default or it is only enabled if routeConfig has the RoutePolicy defined?

Also if the upstream host is down, after how many requests it would be ignored for load balancing? In my testing, it was not removed even after 10 requests have failed. Is Active/Passive health check disabled by default? This host is part of the cluster, server by EDS.

Thanks,

Rama

[Rama Rao]

Update:
My service is grpc service. I tried enabling RetryPolicy on route with retry_on set to 5xx. My client got the following error

java.util.concurrent.ExecutionException: io.grpc.StatusRuntimeException: UNAVAILABLE: HTTP status code 503
invalid content-type: text/plain
headers: Metadata(:status=503,content-length=57,content-type=text/plain,date=Tue, 19 Dec 2017 08:13:13 GMT,server=envoy)
DATA-----------------------------
upstream connect error or disconnect/reset before headers

but the request was not retried even though the status is 503.
Is any other config is required?

[Matt Klein]

Is Request Retry enabled by default or it is only enabled if routeConfig has the RoutePolicy defined?

Not enabled by default. See https://www.envoyproxy.io/docs/envoy/latest/configuration/http_filters/router_filter#x-envoy-retry-on, https://www.envoyproxy.io/docs/envoy/latest/configuration/http_filters/router_filter#x-envoy-retry-grpc-on, etc.

 Also if the upstream host is down, after how many requests it would be ignored for load balancing?

Never by default. See https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/outlier 

--
You received this message because you are subscribed to the Google Groups "envoy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to envoy-users+unsubscribe@googlegroups.com.
To post to this group, send email to envoy...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/envoy-users/8008f24f-eebc-4c7c-ada9-db914362eb01%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Matt Klein

Software Engineer

mkl...@lyft.com

https://calendly.com/mattklein123

[Rama Rao]

Matt,

Thanks. Even if I set the RetryPolicy for the routeConfig for a grpc service, the Retry is not happening as mentioned in my earlier email. Do you see any thing wrong with that output?

[Matt Klein]

I couldn't say without debugging your specific case. Related is https://github.com/envoyproxy/envoy/issues/1934 but that really has nothing to do with retry.

I would look at stats, logging, etc. Retry with gRPC unary calls should work fine. We do this all the time at Lyft.

[Rama Rao]

Ok. Thanks. I have enabled logs. Looks like it is trying to retry. See below. But it is trying to retry to the same host which the original request failed. I think I may have misunderstood the request retry here - i thought it would retry to a different host in the load balancer pool. Looks like retry always happens to the same node. From client perspective this request is a failure. Only way to possibly avoid this failure is by configuring passive health check. Is that a fair statement?

outer] external/envoy/source/common/router/router.cc:706] [C19][S9472752984650552541] performing retry
[2017-12-20 19:35:13.006][45153][debug][connection] external/envoy/source/common/network/connection_impl.cc:526] [C21] connecting to 127.0.0.1:7777
[2017-12-20 19:35:13.007][45153][debug][connection] external/envoy/source/common/network/connection_impl.cc:535] [C21] connection in progress
[2017-12-20 19:35:13.007][45153][debug][http2] external/envoy/source/common/http/http2/codec_impl.cc:625] [C21] setting stream-level initial window size to 268435456
[2017-12-20 19:35:13.007][45153][debug][http2] external/envoy/source/common/http/http2/codec_impl.cc:649] [C21] updating connection-level initial window size to 268435456
[2017-12-20 19:35:13.007][45153][debug][router] external/envoy/source/common/router/router.cc:901] [C19][S9472752984650552541] pool ready

[Matt Klein]

i thought it would retry to a different host in the load balancer pool

The current implementation goes through the normal LB path and can pick the same host again (based on chance and whatever algorithm is being used). Previous hosts are not excluded. This is something I would like to eventually make better.

[Rama Rao]

Ok. Thanks.

[Shannon Coen]

Rama, 

Would you open a github issue for this? 

This looks like a gap we'll need to address before we can replace the Cloud Foundry proxy with Envoy, and we may be able to contribute an enhancement. 

Thank you!

Shannon Coen

Product Manager, Cloud Foundry

Pivotal, Inc.

To view this discussion on the web visit https://groups.google.com/d/msgid/envoy-users/CAFfmCFFbTsmNTzp95W8T3EpiRBcFEYWAiiqt6-o33u%3Dg_VhMLA%40mail.gmail.com.

[Rama Rao]

Created the issue https://github.com/envoyproxy/envoy/issues/2259.

Thanks,

Rama