Streaming Client With keepalive.MaxConnectionAge

[Jeff Nadler]

We have a golang streaming client that connects and sends messages forever.   The server has keepalive.MaxConnectionAge set to 5 minutes to help with load balancing.

Are long-streaming clients and MaxConnectionAge expected to work together?  

What I see in practice is that the underlying connection transparently reconnects (TRANSIENT_FAILURE -> CONNECTING -> READY) at 5 minutes, and everything seems fine, but when we reach MaxConnectionAge + MaxConnectionAgeGrace the client gets an 'EOF' error and fails.

This seems like it _might_ be a bug in grpc-go, but it also might be me trying to combine features in an invalid way?

[Jeff Nadler]

Follow up for posterity:  MaxConnectionAgeGrace doesn't behave the way I'd expected, and the docs aren't as clear as they could be.   I expected MaxConnectionAge to 'ask nicely' (send a GoAway) to close the connection and then MaxConnectionAgeGrace would only forcibly close the connection if RPCs were blocking MaxConnectionAge from taking effect.

That's not what it does in practice.   Even after MaxConnectionAge is successful at triggering the connection to close and be reestablished (which I'd expected to reset the MaxConnectionAgeGrace timer), MaxConnectionAgeGrace hits and forces the connection closed, resulting in an EOF error at the client.