ESPv2 with Firebase Authentication - Jwks remote fetch is failed

858 views
Skip to first unread message

timot...@lynxanalytics.com

unread,
Dec 27, 2019, 5:16:03 AM12/27/19
to Google Cloud Endpoints
Hi all,

I have a ESPv1 which uses firebase authentication that works without problems. I tried to use ESPv2 but it gives a "Jwks remote fetch is failed" error.

To reproduce the problem, I replicated https://cloud.google.com/endpoints/docs/openapi/get-started-cloud-functions but added firebase authentication as in https://cloud.google.com/endpoints/docs/openapi/authenticating-users-firebase and tried to access it by

curl "https://${ENDPOINTS_HOST}/hello" --header "Authorization: Bearer ${TOKEN}"

and it gave the same error.

Any suggestions on fixing the problem? 

Thanks,
Timothy

Jilin Xia

unread,
Dec 27, 2019, 11:36:50 AM12/27/19
to timot...@lynxanalytics.com, Google Cloud Endpoints
Timothy, I will look into this and get back to you ASAP.

Thanks

--
You received this message because you are subscribed to the Google Groups "Google Cloud Endpoints" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/425a370b-5ce8-428b-a159-cd69b5f4c6a3%40googlegroups.com.


--
Regards
Jilin

Jilin Xia

unread,
Dec 27, 2019, 9:31:41 PM12/27/19
to timot...@lynxanalytics.com, Google Cloud Endpoints
Hi, Timothy:

     This is actually an envoy jwt_authn filter issue. It doesn't support x509.


We will try to solve this, either on the Envoy side, or on ESPv2 control plane. 

A short term workaround is instead of using x509 format, please use jwk uri:


Sorry about the inconvenience.

Regards
Jilin
--
Regards
Jilin

Grzegorz Lipecki

unread,
Dec 28, 2019, 3:13:05 AM12/28/19
to Google Cloud Endpoints
Hi Jilin,

with jwk it works as expected.
Thanks for help!

It may be worth to also change uri in guide to avoid misconception for new users (https://cloud.google.com/endpoints/docs/openapi/authenticating-users-firebase).

Regards,
Grzegorz


W dniu sobota, 28 grudnia 2019 03:31:41 UTC+1 użytkownik Jilin Xia napisał:
Hi, Timothy:

     This is actually an envoy jwt_authn filter issue. It doesn't support x509.


We will try to solve this, either on the Envoy side, or on ESPv2 control plane. 

A short term workaround is instead of using x509 format, please use jwk uri:


Sorry about the inconvenience.

Regards
Jilin

On Fri, Dec 27, 2019 at 8:36 AM Jilin Xia <jili...@google.com> wrote:
Timothy, I will look into this and get back to you ASAP.

Thanks

On Fri, Dec 27, 2019 at 2:16 AM timothy.lin via Google Cloud Endpoints <google-clou...@googlegroups.com> wrote:
Hi all,

I have a ESPv1 which uses firebase authentication that works without problems. I tried to use ESPv2 but it gives a "Jwks remote fetch is failed" error.

To reproduce the problem, I replicated https://cloud.google.com/endpoints/docs/openapi/get-started-cloud-functions but added firebase authentication as in https://cloud.google.com/endpoints/docs/openapi/authenticating-users-firebase and tried to access it by

curl "https://${ENDPOINTS_HOST}/hello" --header "Authorization: Bearer ${TOKEN}"

and it gave the same error.

Any suggestions on fixing the problem? 

Thanks,
Timothy

--
You received this message because you are subscribed to the Google Groups "Google Cloud Endpoints" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endpoints+unsub...@googlegroups.com.


--
Regards
Jilin


--
Regards
Jilin

timot...@lynxanalytics.com

unread,
Dec 28, 2019, 4:28:53 AM12/28/19
to Google Cloud Endpoints
Thanks Jilin! Using jwk works.

Regards,
Timothy


On Saturday, December 28, 2019 at 10:31:41 AM UTC+8, Jilin Xia wrote:
Hi, Timothy:

     This is actually an envoy jwt_authn filter issue. It doesn't support x509.


We will try to solve this, either on the Envoy side, or on ESPv2 control plane. 

A short term workaround is instead of using x509 format, please use jwk uri:


Sorry about the inconvenience.

Regards
Jilin

On Fri, Dec 27, 2019 at 8:36 AM Jilin Xia <jili...@google.com> wrote:
Timothy, I will look into this and get back to you ASAP.

Thanks

On Fri, Dec 27, 2019 at 2:16 AM timothy.lin via Google Cloud Endpoints <google-clou...@googlegroups.com> wrote:
Hi all,

I have a ESPv1 which uses firebase authentication that works without problems. I tried to use ESPv2 but it gives a "Jwks remote fetch is failed" error.

To reproduce the problem, I replicated https://cloud.google.com/endpoints/docs/openapi/get-started-cloud-functions but added firebase authentication as in https://cloud.google.com/endpoints/docs/openapi/authenticating-users-firebase and tried to access it by

curl "https://${ENDPOINTS_HOST}/hello" --header "Authorization: Bearer ${TOKEN}"

and it gave the same error.

Any suggestions on fixing the problem? 

Thanks,
Timothy

--
You received this message because you are subscribed to the Google Groups "Google Cloud Endpoints" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endpoints+unsub...@googlegroups.com.


--
Regards
Jilin


--
Regards
Jilin

Teju Nareddy

unread,
Dec 29, 2019, 6:34:07 PM12/29/19
to Google Cloud Endpoints
Thanks for the feedback. We are looking into better documenting these cases for ESPv2. A migration guide from ESP to ESPv2 would be helpful.

To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endpoints+unsubscri...@googlegroups.com.


--
Regards
Jilin


--
Regards
Jilin

Daniel Escobedo

unread,
Jan 2, 2020, 6:45:59 PM1/2/20
to Google Cloud Endpoints
Thanks! This worked for me as well. I agree with Grzegorz about that document he referenced. That's what I followed, and that's how I got this error.


On Friday, December 27, 2019 at 6:31:41 PM UTC-8, Jilin Xia wrote:
Hi, Timothy:

     This is actually an envoy jwt_authn filter issue. It doesn't support x509.


We will try to solve this, either on the Envoy side, or on ESPv2 control plane. 

A short term workaround is instead of using x509 format, please use jwk uri:


Sorry about the inconvenience.

Regards
Jilin

On Fri, Dec 27, 2019 at 8:36 AM Jilin Xia <jili...@google.com> wrote:
Timothy, I will look into this and get back to you ASAP.

Thanks

On Fri, Dec 27, 2019 at 2:16 AM timothy.lin via Google Cloud Endpoints <google-clou...@googlegroups.com> wrote:
Hi all,

I have a ESPv1 which uses firebase authentication that works without problems. I tried to use ESPv2 but it gives a "Jwks remote fetch is failed" error.

To reproduce the problem, I replicated https://cloud.google.com/endpoints/docs/openapi/get-started-cloud-functions but added firebase authentication as in https://cloud.google.com/endpoints/docs/openapi/authenticating-users-firebase and tried to access it by

curl "https://${ENDPOINTS_HOST}/hello" --header "Authorization: Bearer ${TOKEN}"

and it gave the same error.

Any suggestions on fixing the problem? 

Thanks,
Timothy

--
You received this message because you are subscribed to the Google Groups "Google Cloud Endpoints" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endpoints+unsub...@googlegroups.com.


--
Regards
Jilin


--
Regards
Jilin

akee...@gmail.com

unread,
Jan 12, 2020, 6:20:24 PM1/12/20
to Google Cloud Endpoints
I was following similar doc for using Google user authentication with ESPv2 : https://cloud.google.com/endpoints/docs/openapi/authenticating-users-google-id 

It gives the same error mgs. "Jwks remote fetch is failed"

I was able to resolve by adding this line to my openapi-functions.yaml and redeploying :
x-google-jwks_uri: "https://www.googleapis.com/oauth2/v3/certs"

- Akeel

Xuyang(Jason) Tao

unread,
Jan 13, 2020, 1:56:03 PM1/13/20
to Google Cloud Endpoints
Hi Akeel,

Are you using OpenID connection (https://openid.net/connect/  or https://ldapwiki.com/wiki/Openid-configuration  )?  If not, you do need specify
the jwks uri.

Teju Nareddy

unread,
Jan 13, 2020, 3:08:39 PM1/13/20
to Google Cloud Endpoints
Thanks for the feedback. 

For ESPv1: With OpenID connection, you don't need to specify "x-google-jwks_uri". You only need to specify "issuer". Following to the OpenID spec, ESPv1 will try to fetch a JSON from a URI as issuer + ".well-known/openid-configuration" and get its field "jwks_uri" and use it to fetch public key. 

For your use-case with Google ID Tokens, ESPv2 should be fetching the config from https://accounts.google.com/.well-known/openid-configuration. We did some investigation and we found a small bug in ESPv2 that causes this to fail.

I am working on fixing the bug, hopefully the fix will be released in a few days. I will also work on updating the Authentication method docs to document the statements above.

akee...@gmail.com

unread,
Jan 14, 2020, 10:44:55 AM1/14/20
to Google Cloud Endpoints
Wasn't using OpenID connection.
-A

akee...@gmail.com

unread,
Jan 14, 2020, 10:47:03 AM1/14/20
to Google Cloud Endpoints
Thanks Teju! I spun my wheels for a while on this. Will be good to have the docs update.
- A

Teju Nareddy

unread,
Jan 14, 2020, 9:29:30 PM1/14/20
to Google Cloud Endpoints
We clarified the use of "x-google-jwks_uri" in the OpenAPI Extensions documentation. We also updated the Google ID Token documentation to always include "x-google-jwks_uri". Thanks for reporting the issue Akeel!

x509 Support and OpenID Connect Discovery in ESPv2's JWT authentication are fixed, but not released yet. We hope to release the 2 fixes sometime this week or early next week.

Teju Nareddy

unread,
Jan 24, 2020, 2:01:29 PM1/24/20
to Google Cloud Endpoints
FYI: Both bugs have been fixed in the v2.2.0 release on Wednesday.

itg...@gmail.com

unread,
Jan 24, 2020, 9:40:01 PM1/24/20
to Google Cloud Endpoints
Perfect! The docs are spot on now. Thanks again Teju.
- Akeel

TD Gonzales

unread,
Oct 15, 2021, 3:13:18 AM10/15/21
to Google Cloud Endpoints
This problem seems to have come back. 

I was using https://www.googleapis.com/service_accounts/v1/metadata/x509/svc-account@{project_id}.iam.gserviceaccount.com and got around it using the workaround defined above. 

-TD

Alejandro Lorenzo Martin

unread,
Oct 20, 2021, 6:05:44 AM10/20/21
to Google Cloud Endpoints
Hello TD Gonzales,

Did the issue arise from custom configuration, or could you reproduce it like in the first message of this thread? If it was the former, could you provide more details and a reproducible example so we can look into it?

Thanks!

TD Gonzales

unread,
Oct 20, 2021, 5:13:26 PM10/20/21
to Google Cloud Endpoints
After a few days this problem seems to have gone away. I can not reproduce that same problem after reverting to that commit. I am using the x509 endpoint that was mentioned above now without a problem. 

¯\_(ツ)_/¯

Gabriel Panza

unread,
Apr 11, 2022, 2:57:59 PM4/11/22
to Google Cloud Endpoints
Hi, 

I've been using the custom configuration for GRPC as described here https://cloud.google.com/endpoints/docs/grpc/grpc-service-config, still I got the same error:

"Jwks remote fetch is failed"

I'm hosting the jwks.json in a GCS Bucket and already verified that the json is valid and everything.

Any Ideas?

Reply all
Reply to author
Forward
0 new messages