avoid RST on half-close?

[whja...@gmail.com]

It looks like Envoy resets bidirectional gRPC streams when only the client side of the stream closes. Is there a way to configure Envoy so that it doesn't reset http2 streams on half-close?

Here's the example Python code and Envoy config I used for testing:

envoy.yaml: https://pastebin.com/s9ECZRjy

server: https://pastebin.com/VrpwFQYZ

client no-close: https://pastebin.com/ZGPhFZFm

client half-close: https://pastebin.com/4ww0c5y6

grpc api: https://github.com/grpc/grpc/blob/master/examples/protos/route_guide.proto

With the no-close client, I continue to stream forever. With the half-close client, after about 15 seconds I see:

grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.INTERNAL, Received RST_STREAM with error code 0)>

If I take Envoy out of the loop then both the no-close and half-close clients stream forever.

envoy version = v1.7.0

Whitney

[Matt Klein]

Unfortunately I don't think so currently, and I suspect this would be a non-trivial (though appropriate) thing to fix. I would open an issue to track.

I realize it's not a great answer, but the workaround would be to leave the client stream open and indicate "finished" in some application specific way.

--
You received this message because you are subscribed to the Google Groups "envoy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to envoy-users...@googlegroups.com.
To post to this group, send email to envoy...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/envoy-users/bcf5649b-dfe8-4f38-a9e5-e8966a8e6f8e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Matt Klein

Software Engineer

mkl...@lyft.com

https://calendly.com/mattklein123

[whja...@gmail.com]

Ok. Bummer. Thanks for the info. I'll open an issue.

[whja...@gmail.com]

I did find a config change that helps work around this problem. It seems Envoy tears down streams on half-close only after a configurable amount of time has elapsed. The specific config that rules this is route.RouteAction.timeout: https://www.envoyproxy.io/docs/envoy/v1.7.1/api-v2/api/v2/route/route.proto#route-routeaction

The default is 15 seconds. I've adjust to 1 hour for my application. Maybe this will prevent timely cleanup of connection resources and cause other problems. We'll see.

[whja...@gmail.com]

Hi Matt,

I appreciate that this is a non-trivial thing to fix. I wish I were more of C++ programmer and could work on this myself since it comes up frequently in my world. One thing puzzles me though: why is it that this (apparently) doesn't effect lots of Envoy users? If my understanding of this is correct, every gRPC server streaming method that lives for longer than 15 secs and runs through Envoy is bitten by this bug. Many bidirectional gRPC methods run into this too unless clients go out of their way to delay half-closing their side of the bidirectional stream. I would have expected this to be a complete show stopper but from what I can tell, only a few people seem to care. Do you understand why that is?

Also, do you have any sense as to whether this will continue to be a bug that persists in Envoy for a long time? Is it on anyone's radar to fix this?

Thanks,

Whitney

[Matt Klein]

If changing the timeout fixes your issue, I don't think the client is actually resetting their side of the stream.

One thing puzzles me though: why is it that this (apparently) doesn't effect lots of Envoy users? 

I think very few people are using streaming APIs in production currently. I think this will change as these types of APIs are more widely adopted.

 I would have expected this to be a complete show stopper but from what I can tell, only a few people seem to care. Do you understand why that is?

A few reasons:

1. Most people don't care at all.
2. Those that do care have been OK just disabling timeouts completely in many cases.
3. We have now added per-stream idle timeout, which is what you really want for streaming cases. So I would recommend you configure streaming APIs with 0 request timeout, but a non-zero stream idle timeout.

 Also, do you have any sense as to whether this will continue to be a bug that persists in Envoy for a long time? Is it on anyone's radar to fix this?

If we are now talking about the originally reported client side reset issue, I don't know that anyone has expressed any interest in fixing this, nor that it matters for any concrete use case that I know. 

--

You received this message because you are subscribed to the Google Groups "envoy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to envoy-users...@googlegroups.com.
To post to this group, send email to envoy...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/envoy-users/cb1167ea-6f4a-4b35-a513-cab79836a488%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[whja...@gmail.com]

Thanks for the info and advice. I'll play around with a 0 request timeout (I didn't realize that was option) and a non-zero idle timeout.

> If changing the timeout fixes your issue, I don't think the client is actually resetting their side of the stream.

Correct. The gRPC client doesn't send an RST_STREAM frame on half-close. It does set the END_STREAM flag though. Timeout seconds later Envoy sends the client an RST_STREAM frame.

Given the updated documentation for the timeout setting, I guess it's not really fair to call this a bug. I'd argue the behavior is a surprising default but you can disable it and it works as documented. Want me to close the github issue?

[Matt Klein]

Want me to close the github issue?

Sure. Any documentation update PRs appreciated that would have helped you understand this better... 

--

You received this message because you are subscribed to the Google Groups "envoy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to envoy-users...@googlegroups.com.
To post to this group, send email to envoy...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/envoy-users/a4a08014-6d4e-4c2e-a956-7e7cdec3e45a%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.