rate limit on jwt auth tokens?

86 views
Skip to first unread message

gce...@lacrossetechnology.com

unread,
Jun 18, 2018, 11:43:52 PM6/18/18
to Firebase Google Group
Since last thursday we are getting massive amounts of errors with the JWT service url for firebase Auth on our App Engine Application.  We figured out today that cycling instances so they get new IP's appears to alleviate this issue.  However the question is why would this be an issue in the first place. 

The error we see in the app engine logs are 

Cannot retrive the JWKS json from https://www.googleapis.com/robot/v1/metadata/x509/secur...@system.gserviceaccount.com .  It keeps timeout randomly. But again, if we cycle the instances in App Engine so they are 'new' machines they will work for awhile.

Does firebase have some undocumented limit on JWT token requests??

Hiranya Jayathilaka

unread,
Jun 19, 2018, 1:13:05 PM6/19/18
to fireba...@googlegroups.com
Sounds more like a transient issue with the App Engine networking stack. In any case, how are you accessing the above URL? Are you not caching the response?

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-tal...@googlegroups.com.
To post to this group, send email to fireba...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/56e0bd77-65d7-4246-9843-374e189f8ef3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


--

Hiranya Jayathilaka | Software Engineer | h...@google.com | 650-203-0128

gce...@lacrossetechnology.com

unread,
Jun 19, 2018, 3:25:28 PM6/19/18
to Firebase Google Group
Thanks for the response.  Yes, we have a ticket created with Google Cloud Support on the App Engine side just in case it is something to do with networking (DNS) over there.  We are caching the response where we can and have followed the guide to integrate this Firebase Service to App Engine precisely.  Plus, the application stack has worked fine for the past 6 months overall with about 75% of peak load experienced during that period.

We have had spikes in usage the past few weeks, as we are implementing new systems.  It is only during these periods we see these issues.  Today load is down to 'normal' and we aren't seeing hte issues.  Seems to behave use like a rate limit.  Will follow up with this thread when i hear more from Google Cloud support.


On Tuesday, June 19, 2018 at 12:13:05 PM UTC-5, Hiranya Jayathilaka wrote:
Sounds more like a transient issue with the App Engine networking stack. In any case, how are you accessing the above URL? Are you not caching the response?

On Mon, Jun 18, 2018 at 8:43 PM <gce...@lacrossetechnology.com> wrote:
Since last thursday we are getting massive amounts of errors with the JWT service url for firebase Auth on our App Engine Application.  We figured out today that cycling instances so they get new IP's appears to alleviate this issue.  However the question is why would this be an issue in the first place. 

The error we see in the app engine logs are 

Cannot retrive the JWKS json from https://www.googleapis.com/robot/v1/metadata/x509/securetoken@system.gserviceaccount.com .  It keeps timeout randomly. But again, if we cycle the instances in App Engine so they are 'new' machines they will work for awhile.


Does firebase have some undocumented limit on JWT token requests??

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-tal...@googlegroups.com.
To post to this group, send email to fireba...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/56e0bd77-65d7-4246-9843-374e189f8ef3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Hiranya Jayathilaka

unread,
Jun 19, 2018, 4:52:42 PM6/19/18
to fireba...@googlegroups.com
The response from that endpoint is cacheable up to 24 hours, if the HTTP cache-control protocol is followed. 

On Tue, Jun 19, 2018 at 12:25 PM <gce...@lacrossetechnology.com> wrote:
Thanks for the response.  Yes, we have a ticket created with Google Cloud Support on the App Engine side just in case it is something to do with networking (DNS) over there.  We are caching the response where we can and have followed the guide to integrate this Firebase Service to App Engine precisely.  Plus, the application stack has worked fine for the past 6 months overall with about 75% of peak load experienced during that period.

We have had spikes in usage the past few weeks, as we are implementing new systems.  It is only during these periods we see these issues.  Today load is down to 'normal' and we aren't seeing hte issues.  Seems to behave use like a rate limit.  Will follow up with this thread when i hear more from Google Cloud support.


On Tuesday, June 19, 2018 at 12:13:05 PM UTC-5, Hiranya Jayathilaka wrote:
Sounds more like a transient issue with the App Engine networking stack. In any case, how are you accessing the above URL? Are you not caching the response?

On Mon, Jun 18, 2018 at 8:43 PM <gce...@lacrossetechnology.com> wrote:
Since last thursday we are getting massive amounts of errors with the JWT service url for firebase Auth on our App Engine Application.  We figured out today that cycling instances so they get new IP's appears to alleviate this issue.  However the question is why would this be an issue in the first place. 

The error we see in the app engine logs are 

Cannot retrive the JWKS json from https://www.googleapis.com/robot/v1/metadata/x509/secur...@system.gserviceaccount.com .  It keeps timeout randomly. But again, if we cycle the instances in App Engine so they are 'new' machines they will work for awhile.


Does firebase have some undocumented limit on JWT token requests??

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-tal...@googlegroups.com.
To post to this group, send email to fireba...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/56e0bd77-65d7-4246-9843-374e189f8ef3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


--

Hiranya Jayathilaka | Software Engineer | h...@google.com | 650-203-0128

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-tal...@googlegroups.com.
To post to this group, send email to fireba...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

gce...@lacrossetechnology.com

unread,
Jun 19, 2018, 6:17:29 PM6/19/18
to Firebase Google Group
We aren't making the call in our code.  It's done by internal libraries (espauthenticator) from google.  Unless there is a configuration to tweak it in that, we cache it in elixir but it's not our code doing authentication in App Engine.  


On Tuesday, June 19, 2018 at 3:52:42 PM UTC-5, Hiranya Jayathilaka wrote:
The response from that endpoint is cacheable up to 24 hours, if the HTTP cache-control protocol is followed. 

On Tue, Jun 19, 2018 at 12:25 PM <gce...@lacrossetechnology.com> wrote:
Thanks for the response.  Yes, we have a ticket created with Google Cloud Support on the App Engine side just in case it is something to do with networking (DNS) over there.  We are caching the response where we can and have followed the guide to integrate this Firebase Service to App Engine precisely.  Plus, the application stack has worked fine for the past 6 months overall with about 75% of peak load experienced during that period.

We have had spikes in usage the past few weeks, as we are implementing new systems.  It is only during these periods we see these issues.  Today load is down to 'normal' and we aren't seeing hte issues.  Seems to behave use like a rate limit.  Will follow up with this thread when i hear more from Google Cloud support.


On Tuesday, June 19, 2018 at 12:13:05 PM UTC-5, Hiranya Jayathilaka wrote:
Sounds more like a transient issue with the App Engine networking stack. In any case, how are you accessing the above URL? Are you not caching the response?

On Mon, Jun 18, 2018 at 8:43 PM <gce...@lacrossetechnology.com> wrote:
Since last thursday we are getting massive amounts of errors with the JWT service url for firebase Auth on our App Engine Application.  We figured out today that cycling instances so they get new IP's appears to alleviate this issue.  However the question is why would this be an issue in the first place. 

The error we see in the app engine logs are 

Cannot retrive the JWKS json from https://www.googleapis.com/robot/v1/metadata/x509/securetoken@system.gserviceaccount.com .  It keeps timeout randomly. But again, if we cycle the instances in App Engine so they are 'new' machines they will work for awhile.


Does firebase have some undocumented limit on JWT token requests??

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-tal...@googlegroups.com.
To post to this group, send email to fireba...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/56e0bd77-65d7-4246-9843-374e189f8ef3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


--

Hiranya Jayathilaka | Software Engineer | h...@google.com | 650-203-0128

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-tal...@googlegroups.com.
To post to this group, send email to fireba...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/49519c27-710e-485e-9271-56febbdeaee1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply all
Reply to author
Forward
0 new messages