ESPv2 gRPC Support

350 views
Skip to first unread message

Prateek Malhotra

unread,
Jan 10, 2020, 4:37:05 PM1/10/20
to Google Cloud Endpoints
Hello,

According to the ESPv2 repository - ESPv2 should support gRPC:

Google Cloud Platform ESPv2 is a service proxy which enables API management capabilities for JSON/REST or gRPC API services using Google Service Infrastructure. The current implementation uses Envoy as a service proxy.

I know there has been discussion of gRPC on Cloud Run here: https://github.com/cloudendpoints/esp/issues/742

I wanted to move the discussion here in hopes of gaining an understanding of whether or not ESPv2 actually does support gRPC at this time (on or off of Cloud Run), and if not, when it is planned to be implemented.

In my testing, I've been able to get ESPv2 to accept gRPC requests, but it doesn't seem to be routing them to the backend correctly. My backend is protected via IAP + TLS and ESPv1 seems to support this without issue.

I used the gcloud image build script to deploy my ESP image and provide the following arguments:

--log_request_headers=true
--enable_debug
--tracing_sample_rate=1
--tracing_incoming_context=x-cloud-trace-context
--tracing_outgoing_context=traceparent
--backend_protocol=grpc

The log output shows that ESPv2 can accept gRPC requests, authenticate them appropriately, and try to contact the backend. From the log output below, it looks like it's falling to add Bearer authentication for IAP to the backend request and maybe having trouble connecting over TLS, but otherwise looks to support gRPC and does include the `x-endpoint-api-userinfo` header.

Here is the output of some of the applicable debug logs (highlighted to show indications of the requests progress): 

23:21:32.803 27 envoy] [27][http][external/envoy/source/common/http/conn_manager_impl.cc:708] [C24][S12315280653272850522] request headers complete (end_stream=false):\n\':authority\', \'<my-service>-uc.a.run.app\'\n\':path\', \'/service.v1alpha.Interop/MutateFiles\'\n\':method\', \'POST\'\n\':scheme\', \'http\'\n\'content-type\', \'application/grpc\'\n\'user-agent\', \'grpc-go/1.26.0\'\n\'authorization\', \'Bearer <TOKEN-REMOVED>\'\n\'x-cloud-trace-context\', \'d610a57b2025d586dc2a0e0b45cda9c1/10536684767275038589;o=1\'\n\'x-forwarded-for\', \'<my-ip-removed>\'\n\'x-forwarded-proto\', \'https\'\n\'forwarded\', \'for=\"<my-ip-removed>\";proto=https\'\n\'content-length\', \'119\'\n\'te\', \'trailers\'
23:21:32.803 27 envoy] [27][jwt][external/envoy/source/extensions/filters/http/jwt_authn/filter.cc:124] Called Filter : setDecoderFilterCallbacks
23:21:32.803 27 envoy] [27][filter][src/envoy/http/path_matcher/filter.cc:54] matched operation: service.v1alpha.Interop.MutateFiles
23:21:32.803 27 envoy] [27][jwt][external/envoy/source/extensions/filters/http/jwt_authn/filter.cc:46] Called Filter : decodeHeaders
23:21:32.803 27 envoy] [27][jwt][bazel-out/k8-opt/bin/external/envoy/source/extensions/filters/http/jwt_authn/_virtual_includes/filter_config_interface/extensions/filters/http/jwt_authn/filter_config.h:117] use filter state value service.v1alpha.Interop.MutateFiles to find verifier.
23:21:32.803 27 envoy] [27][jwt][external/envoy/source/extensions/filters/http/jwt_authn/authenticator.cc:107] JWT authentication starts (allow_failed=false)
23:21:32.803 27 envoy] [27][jwt][external/envoy/source/extensions/filters/http/jwt_authn/authenticator.cc:127] Verifying JWT token of issuer api-signer@<project-id>.iam.gserviceaccount.com
23:21:32.804 27 envoy] [27][jwt][external/envoy/source/extensions/filters/http/jwt_authn/authenticator.cc:245] JWT token verification completed with: OK
23:21:32.804 27 envoy] [27][jwt][external/envoy/source/extensions/filters/http/jwt_authn/filter.cc:84] Called Filter : check complete OK
23:21:32.804 27 envoy] [27][filter][src/envoy/http/service_control/filter.cc:46] Called ServiceControl Filter : decodeHeaders
23:21:32.804 27 envoy] [27][filter][src/envoy/http/backend_auth/filter.cc:56] Found operation: service.v1alpha.Interop.MutateFiles
23:21:32.804 27 envoy] [27][filter][src/envoy/http/backend_routing/filter.cc:42] Found operation: service.v1alpha.Interop.MutateFiles
23:21:32.804 27 envoy] [27][filter][src/envoy/http/backend_routing/filter.cc:46] No backend routing rule found for operation service.v1alpha.Interop.MutateFiles
23:21:32.804 27 envoy] [27][router][external/envoy/source/common/router/router.cc:438] [C24][S12315280653272850522] cluster \'<my-service>-uc.a.run.app_local\' match for URL \'/service.v1alpha.Interop/MutateFiles\'
23:21:32.804 27 envoy] [27][router][external/envoy/source/common/router/router.cc:553] [C24][S12315280653272850522] router decoding headers:\n\':authority\', \'<my-service>-uc.a.run.app\'\n\':path\', \'/service.v1alpha.Interop/MutateFiles\'\n\':method\', \'POST\'\n\':scheme\', \'http\'\n\'content-type\', \'application/grpc\'\n\'user-agent\', \'grpc-go/1.26.0\'\n\'x-cloud-trace-context\', \'d610a57b2025d586dc2a0e0b45cda9c1/10536684767275038589;o=1\'\n\'x-forwarded-for\', \'<my-ip-removed>\'\n\'x-forwarded-proto\', \'https\'\n\'forwarded\', \'for=\"<my-ip-removed>\";proto=https\'\n\'content-length\', \'119\'\n\'te\', \'trailers\'\n\'x-request-id\', \'cf477066-b418-914d-8a3e-c0a624219021\'\n\'x-endpoint-api-userinfo\', \'<REMOVED>>\'\n\'x-envoy-expected-rq-timeout-ms\', \'15000\'\n\'traceparent\', \'00-d610a57b2025d586dc2a0e0b45cda9c1-98bdad355f38e5b7-01\'
23:21:32.804 27 envoy] [27][client][external/envoy/source/common/http/codec_client.cc:31] [C29] connecting
23:21:32.804 27 envoy] [27][connection][external/envoy/source/common/network/connection_impl.cc:707] [C29] connecting to 127.0.0.1:8082
23:21:32.804 27 envoy] [27][connection][external/envoy/source/common/network/connection_impl.cc:716] [C29] connection in progress
23:21:32.804 27 envoy] [27][http2][external/envoy/source/common/http/http2/codec_impl.cc:912] [C29] setting stream-level initial window size to 268435456
23:21:32.804 27 envoy] [27][http2][external/envoy/source/common/http/http2/codec_impl.cc:934] [C29] updating connection-level initial window size to 268435456
23:21:32.804 27 envoy] [27][pool][external/envoy/source/common/http/conn_pool_base.cc:20] queueing request due to no available connections
23:21:32.804 27 envoy] [27][http][external/envoy/source/common/http/conn_manager_impl.cc:1250] [C24][S12315280653272850522] request end stream
23:21:32.804 27 envoy] [27][jwt][external/envoy/source/extensions/filters/http/jwt_authn/filter.cc:108] Called Filter : decodeData
23:21:32.804 27 envoy] [27][filter][src/envoy/http/service_control/filter.cc:100] Called ServiceControl Filter : decodeData
23:21:32.804 27 envoy] [27][connection][external/envoy/source/common/network/connection_impl.cc:564] [C29] delayed connection error: 111
23:21:32.804 27 envoy] [27][connection][external/envoy/source/common/network/connection_impl.cc:189] [C29] closing socket: 0
23:21:32.804 27 envoy] [27][client][external/envoy/source/common/http/codec_client.cc:88] [C29] disconnect. resetting 0 pending requests
23:21:32.804 27 envoy] [27][pool][external/envoy/source/common/http/http2/conn_pool.cc:152] [C29] client disconnected
23:21:32.804 27 envoy] [27][router][external/envoy/source/common/router/router.cc:914] [C24][S12315280653272850522] upstream reset: reset reason connection failure
23:21:32.804 27 envoy] [27][http][external/envoy/source/common/http/conn_manager_impl.cc:1347] [C24][S12315280653272850522] Sending local reply with details upstream_reset_before_response_started{connection failure}
23:21:32.804 27 envoy] [27][filter][src/envoy/http/service_control/filter.cc:122] Called ServiceControl Filter : encodeHeaders before
23:21:32.804 27 envoy] [27][http][external/envoy/source/common/http/conn_manager_impl.cc:1545] [C24][S12315280653272850522] encoding headers via codec (end_stream=true):\n\':status\', \'200\'\n\'content-type\', \'application/grpc\'\n\'grpc-status\', \'14\'\n\'grpc-message\', \'upstream connect error or disconnect/reset before headers. reset reason: connection failure\'\n\'date\', \'Thu, 09 Jan 2020 23:21:32 GMT\'\n\'server\', \'envoy\'
23:21:32.804 27 envoy] [27][http2][external/envoy/source/common/http/http2/codec_impl.cc:742] [C24] stream closed: 0
23:21:32.804 27 envoy] [27][jwt][external/envoy/source/extensions/filters/http/jwt_authn/filter.cc:39] Called Filter : onDestroy
23:21:32.804 27 envoy] [27][filter][src/envoy/http/service_control/filter.cc:38] Called ServiceControl Filter : onDestroy
23:21:32.804 27 envoy] [27][pool][external/envoy/source/common/http/http2/conn_pool.cc:174] [C29] destroying primary client
23:21:32.804 27 envoy] [27][filter][src/envoy/http/service_control/filter.cc:145] Called ServiceControl Filter : log

If anyone has an idea on how to make further progress on this or any details on gRPC support for ESPv2, please let me know!

Thank you,

Prateek 

Jilin Xia

unread,
Jan 13, 2020, 9:26:16 AM1/13/20
to Prateek Malhotra, Google Cloud Endpoints

Thanks for the feedback, Prateek.

gRPC is not supported yet for ESPv2 Beta, but definitely this is a required feature for GA.

For the error you got, it is upstream connection error,  we can fix it in the next release, so at least  Envoy can connect with your backend. but again, gRPC feature is not officially supported, it may miss some features of ESPv1. 

Regards 
Jilin

--
You received this message because you are subscribed to the Google Groups "Google Cloud Endpoints" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/f95d8d2c-4e7f-402c-838c-d1c169190b43%40googlegroups.com.
--
Regards
Jilin

Teju Nareddy

unread,
Jan 13, 2020, 7:33:57 PM1/13/20
to Google Cloud Endpoints
We are currently working on debugging the error message. We opened an issue with Envoy that you can follow if you are interested: https://github.com/envoyproxy/envoy/issues/9665

To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endpoints+unsub...@googlegroups.com.
--
Regards
Jilin

Prateek Malhotra

unread,
Jan 15, 2020, 10:50:08 AM1/15/20
to Teju Nareddy, Google Cloud Endpoints
Thanks for the updates! It looks like you all figured out what was causing the problem, looking forward to a fix on ESPv2!

I should also note it looks like you all are discussing the TLS issue, but from the logs I've shown, there is still missing authentication for the IAP protected resource:

23:21:32.804 27 envoy] [27][router][external/envoy/source/common/router/router.cc:553] [C24][S12315280653272850522] router decoding headers:
':authority', '<my-service>-uc.a.run.app'
':path', '/service.v1alpha.Interop/MutateFiles'
':method', 'POST'
':scheme', 'http'
'content-type', 'application/grpc'
'user-agent', 'grpc-go/1.26.0'
'x-cloud-trace-context', 'd610a57b2025d586dc2a0e0b45cda9c1/10536684767275038589;o=1'
'x-forwarded-for', '<my-ip-removed>'
'x-forwarded-proto', 'https'
'forwarded', 'for=\"<my-ip-removed>\";proto=https'
'content-length', '119'
'te', 'trailers'
'x-request-id', 'cf477066-b418-914d-8a3e-c0a624219021'
'x-endpoint-api-userinfo', '<REMOVED>>'
'x-envoy-expected-rq-timeout-ms', '15000'
'traceparent', '00-d610a57b2025d586dc2a0e0b45cda9c1-98bdad355f38e5b7-01'

Notice there is no 'authorization', 'Bearer <token>' entry. Will this be supported in ESPv2 and if so, is it on your radar? I assume it can be supported as you all were able to inject the 'trace' and  'x-endpoint-api-userinfo' headers.

Thank you for all your hard work and help on this matter! If I can help in anyway, including testing, please let me know.

Prateek Malhotra


To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endp...@googlegroups.com.
--
Regards
Jilin

--
You received this message because you are subscribed to the Google Groups "Google Cloud Endpoints" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/06f4888d-a962-4cea-b6c9-22a6e7198251%40googlegroups.com.

Wayne Zhang

unread,
Jan 15, 2020, 12:22:58 PM1/15/20
to Google Cloud Endpoints
Yes,  ID token will be added by ESP if your "backend" specified "audience" field.  Normally, "audience" should be the URL to your backend address.  We are working on the gRPC support, once it is ready we will update the doc on how to config it.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endpoints+unsub...@googlegroups.com.
--
Regards
Jilin

--
You received this message because you are subscribed to the Google Groups "Google Cloud Endpoints" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endpoints+unsub...@googlegroups.com.

Teju Nareddy

unread,
Jan 23, 2020, 1:59:31 PM1/23/20
to Google Cloud Endpoints
Hi Prateek, just wanted to give you a quick update on gRPC support in ESPv2 Beta for Serverless.

We have a working prototype for gRPC-JSON transcoding (client is HTTP, server is gRPC). We are still working on gRPC pass-through (client is gRPC, server is gRPC).

Keep in mind that Cloud Run only supports unary gRPC methods. Streaming gRPC support in ESPv2 Beta for Serverless is not planned yet.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endpoints+unsubscri...@googlegroups.com.
--
Regards
Jilin

--
You received this message because you are subscribed to the Google Groups "Google Cloud Endpoints" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endpoints+unsubscri...@googlegroups.com.

Prateek Malhotra

unread,
Jan 23, 2020, 3:05:08 PM1/23/20
to Teju Nareddy, Google Cloud Endpoints
Hi Teju,

I appreciate the update! I saw the 2.2.0 beta release notes and was glad to see some mention of gRPC though as you said its for gRPC-JSON transcoding. It was not immediately clear to me if this resolved the TLS issue or the auth issue I encountered and since the code changes aren't on GitHub I cannot confirm it myself without testing. Unless you can confirm that one, both, or none of these have been addressed beforehand, I will eventually get around to testing it again.

I'm aware of the unary restriction and that's fine for me!

Thank you,

Prateek Malhotra


To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endp...@googlegroups.com.
--
Regards
Jilin

--
You received this message because you are subscribed to the Google Groups "Google Cloud Endpoints" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endp...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Google Cloud Endpoints" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/14283e00-e989-4f5a-9dbb-7512ab683ec9%40googlegroups.com.

Wayne Zhang

unread,
Jan 23, 2020, 3:11:45 PM1/23/20
to Prateek Malhotra, Teju Nareddy, Google Cloud Endpoints
 The key change is to enable the alpn protocol in envoy cluster.  2.2.0 did not have that change.   It should be in the next release.   With that change, only Http to grpc transcoding works,  grpc pass through still doesn't work.  Somehow grpc-status trailer header is dropped in the way.


To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/CAB880WmXLYjfgZ4rQMHdY_%2Bb2%3DZLLJhjkeV4%2B_YZ1_RO3bskAg%40mail.gmail.com.

tao...@google.com

unread,
Jan 28, 2020, 3:18:59 PM1/28/20
to Google Cloud Endpoints
Hi Prateek,

The latest released image gcr.io/endpoints-release/endpoints-runtime-serverless:2.3.0 has supported grpc backend but it works for json downstream request. 
For grpc pass through, we have figured it out and hopefully it would be rolled out soon.



To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endpoints+unsub...@googlegroups.com.
--
Regards
Jilin

--
You received this message because you are subscribed to the Google Groups "Google Cloud Endpoints" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endpoints+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Google Cloud Endpoints" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endpoints+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/14283e00-e989-4f5a-9dbb-7512ab683ec9%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Google Cloud Endpoints" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endpoints+unsub...@googlegroups.com.

Teju Nareddy

unread,
Feb 7, 2020, 3:37:07 PM2/7/20
to Google Cloud Endpoints
Let's continue the discussion here: https://groups.google.com/g/google-cloud-endpoints/c/CRWvA_xQ1yw

To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endpoints+unsubscri...@googlegroups.com.
--
Regards
Jilin

--
You received this message because you are subscribed to the Google Groups "Google Cloud Endpoints" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endpoints+unsubscri...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Google Cloud Endpoints" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endpoints+unsubscri...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/14283e00-e989-4f5a-9dbb-7512ab683ec9%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Google Cloud Endpoints" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endpoints+unsubscri...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages