Proper error handling for streams. Should connections need to be thrown away on some errors?

[jdw...@gmail.com]

Howdy,

I'm trying to build some configuration distribution with gRPC for prefab.cloud. All is working, but I could use a second opinion on whether I'm implementing error handling correctly. 

My clients would like to setup a connection and then listen forever. So far I have found that they successfully reconnect after network blips, but when I deploy the server my clients will hang unless I throw away the connection. Is that expected? 

My approach has been to treat all errors as semi-fatal, back off myself, nuke the connection and start over.

I have some open source client libraries https://github.com/prefab-cloud/prefab-cloud-ruby and https://github.com/prefab-cloud/prefab-cloud-java 

Relevant code is in 

https://github.com/prefab-cloud/prefab-cloud-ruby/blob/master/lib/prefab/config_client.rb#L119-L133

https://github.com/prefab-cloud/prefab-cloud-ruby/blob/master/lib/prefab/client.rb#L91-L94

Does that seem like the right approach? 

(For bonus points you can see my kinda weird "stub" cache. That felt... weird...   but I was trying to avoid creating new stubs everywhere. Am I being silly?)

Thanks!

-Jeff

[Penn (Dapeng) Zhang]

Can you provide more details about what is happening when you deploy the server? And what your client is doing when you deploy the server, like is the client having an ongoing RPC, or is it idle? 

[jdw...@ezcater.com]

Hi. The clients are trying to just keep an everlasting connection to the server (ie I never call onComplete() on the server). 

Deploys are run by AWS ECS. All the instances are in an AWS NLB, then ECS starts removing them from the LB after it has brought new nodes up. I presume that that means the services just disappear into the ether from the perspective of the client. 

I will have to spend some time to go reproduce the errors I'd had. My question here is just whether what I've ended up with is "expected bahavior", specifically that after the server instance I've connected to disappears I should recreate the stub & the channel as well.