Any success with GRPC and the new AWS Application Load Balancer?

[d...@pexlabs.com]

Has anyone had any luck using Amazon's new ALB with GRPC? We are trying to deploy a grpc-java app and have run into a few major issues:

1) With plaintext GRPC on the backend, ALB fails because it tries to upgrade from HTTP/1.1 which grpc-java does not support.

2) With TLS GRPC on the backend, ALB seems to fail for the same reason (TLS/ALPN negotiation succeeds but protocol is not HTTP/2).

I don't see a way to configure ALB to assume HTTP/2 on the backend, so I am at a loss for a reasonable workaround.

On the other hand, fixing grpc-java to support plaintext upgrade seems non trivial as well, although I may take a crack

at that next.

Any suggestions would be most appreciated.

- David

[Eric Anderson]

On Sat, Sep 3, 2016 at 6:16 PM, <d...@pexlabs.com> wrote:

1) With plaintext GRPC on the backend, ALB fails because it tries to upgrade from HTTP/1.1 which grpc-java does not support.

Yep. It isn't supported. The first request being HTTP/1 makes things... weird.

For those following along at home, this issue is now tracking plaintext upgrade.

 

2) With TLS GRPC on the backend, ALB seems to fail for the same reason (TLS/ALPN negotiation succeeds but protocol is not HTTP/2).

That sounds like it doesn't support HTTP/2. I wouldn't be surprised if it didn't support HTTP/2 to the backend. Clients could be HTTP/2, but then it would convert it to HTTP/1.1. This is the same state as with Google Cloud Load Balancer and nginx. Amazon's blog post also doesn't make it clear.

Since TLS doesn't work, I'd assume it doesn't support plaintext as well; if you stand up a test plaintext HTTP/1.1 server you won't see an "Upgrade: h2c" header.

[d...@pexlabs.com]

 

2) With TLS GRPC on the backend, ALB seems to fail for the same reason (TLS/ALPN negotiation succeeds but protocol is not HTTP/2).

That sounds like it doesn't support HTTP/2. I wouldn't be surprised if it didn't support HTTP/2 to the backend. Clients could be HTTP/2, but then it would convert it to HTTP/1.1. This is the same state as with Google Cloud Load Balancer and nginx. Amazon's blog post also doesn't make it clear.

Yup, an AWS support rep just confirmed that ALB indeed does not support HTTP/2 to the backend. I also pointed out that this should be stated more clearly in the docs.

Perhaps someday end-to-end HTTP/2 will become a reality :)

[mvn...@gmail.com]

I'm very interested in this as well.

[shlo...@walkme.com]

David, It's interesting how you finally solved it, did you used ELB instead?

[jdw...@gmail.com]

I successfully got GRPC working in AWS with the (fairly new) NetworkLoadBalancer for http://prefab.cloud

2 main things to know:
1) NLB can’t terminate SSL so you’ll have to do that yourself instead of nice simple AWS very management.

2) you need to run something next to your GRPC server that will listen and 200 to the NLB health checks

For me that was running dropwizard as a wrapper for my GRPC server.

[shlo...@walkme.com]

Ok, thanks for the info.

I'm using now the classic load balancer (ELB) with TCP listener to the gRPC port, it works but sometimes (unexpectedly) i'm getting DEADLINE EXCEED errors and the only way to solve it is by restart both ECS tasks (i'm using gRPC to connect between my two microservices)

Any idea why?

[jdw...@gmail.com]

I’m not seeing that exactly, but i’m Having issues on deployments. Seems like connections don’t reconnect and i’m struggling to feel good about my error handling and connection resetting code. I’m going ruby client to have server. You?

[shlo...@walkme.com]

i'm using node.js one, we fixed something recently with error handling and looks like the problem disappears, praying to god that it's behind us..