The request was aborted because there was no available instance

[Supriya Aggarwal]

Hi,

I started getting this error on one of my cloud functions 2 days back: The request was aborted because there was no available instance.

Does anyone know why this happens? I tried reducing the cold start time by keeping the function warm but that isn't helping either.

And I can see that 3 instances were active for this function at a point in time.

Attaching screenshot.

Thanks,

Supriya.

[Robbert]

Hi,

I'm having the exact same thing on 2 separate Firebase projects, starting today out of the blue. I did not experience these before.

Cheers,

Robbert

[Oskar Helling]

Same here. Never seen this before. The message started appearing a couple of days ago. There is absolutely no possibility the message is due to a spike in demand - it is a dev/test site with only a handful of users.

[Supriya Aggarwal]

:(

We tried keeping the functions warm but that doesn't seem to be working either. Did you try any workarounds?

[Supriya Aggarwal]

Same with us, not a lot of users.

[Oskar Helling]

I updated the firebase-functions -library in package.json which was not quite the latest, but since then I have still received the error message.

I am out of options, but this is huge problem - I can't have my Cloud Functions bailing out on me randomly.

/o

[Daniel Lee]

Hey folks,

Check out https://issuetracker.google.com/issues/153207649#comment3 for an official response from the Google Cloud Functions support team.

tl;dr GCF nodejs runtime used to silently drop requests when instance couldn't be scaled fast enough to respond to demand. Now it's logging the failed request on your project's log, hence the sudden appearance of the issue (release note). For event-triggered functions, this error is usually handled gracefully by automatic retry mechanism in the GCF infrastructure. The same can't be said of HTTP-triggered functions, and the request would have been dropped by the client unless a retry mechanism was already implemented.

If you have any project-specific issue you need help with, please consider reaching out to GCP support.

Daniel

[Robbert]

Hi Daniel, thank you for these insights!

However, it does raise a few more questions:

- If this was released July 7, why is this only affecting us now?

- Since the errors started showing for us in the logs (roughly 30 hours ago), we've also been experiencing errors from these HTTP functions in our clients, similar to what you're describing (no retry mechanism). However, we did not have these errors before. So somehow it seems the errors are still new, and not just the logging of the errors.

Any clue?

Cheers,

Robbert

[Supriya Aggarwal]

Thanks for this, Daniel. But it looks like there was some graceful handling of HTTP functions as well earlier because we never faced data loss issues in the past but we are facing them now, which is what led us to discover this error in the first place. 

Can others confirm/deny this, please?

[Daniel Lee]

1) GCF team initially rolled out this "feature" for python runtime only. They recently enabled the feature for Node runtime sometime last week, hence here we are today.

2) AFAIK, there was never a graceful handling of failed HTTP functions in the GCF infrastructure - the recommendation is to setup retry logic on the client side code. If you are seeing elevated error rate from your HTTP functions, definitely reach out to GCP support. I'm not aware of any widespread outage recognized by the GCF team that would explain what's being reported here.

[Daniel Lee]

Another relevant public issue tracker from the GCF support team: https://issuetracker.google.com/issues/194948300#comment3

[Robbert]

Thank you. For us, there is pretty clear starting point, which was around Aug 30 12:00 pm CEST, and we've been experiencing a relatively low but steady rate of errors ever since.

If it was rolled out last week, wouldn't it be weird to suddenly start just yesterday? We didn't do any deploys.

For us, it's definitely not just the "logging" feature that got enabled. We've been getting errors on the client from our failed HTTP functions matching these aborted request errors. I don't think this is an elevated error rate specifically on our HTTP functions, because the same error happens on all functions, but the only place where we actually notice the effect is HTTP functions because there is no automatic retry. It seems a similar thing is happening for Supriya, because of the HTTP functions suddenly failing.

Just to clarify:

Our old situation (before Aug 30 around 12:00pm CEST):

- no logged errors for firestore/pubsub functions

- no logged errors for HTTP functions

- no client errors from failed HTTP requests

Our new situation (after Aug 30 around 12:00pm CEST):

- logged errors for firestore/pubsub functions
- logged errors for HTTP functions

- client errors from failed HTTP requests

(on 2 separate projects, both not that many users)

I'm worried the recently added feature to log these kind of errors is now obscuring an issue that is actually causing us to have these errors. Did perhaps the allowed startup time for cold starts decrease?

Cheers,

Robbert

[Naser Samara]

Hi

I am facing the same error, it started since Aug 30,2021. (europe-west1).. should I  setup retry logic on the client side code?

thank you

[Supriya Aggarwal]

Yes Robbert, I am facing something very similar. 

I even tried keeping the functions warm but that didn't help :(

[Daniel Lee]

Please do reach out to support! It would be immensely helpful if you can provide them with any relevant metrics and specific time windows where you've seen a rise in error rates for HTTP function invocations.

Due to the large influx of support cases affecting event triggers, your support agent might miscategorized your problem as yet another event-triggered issue. I'm asking for your patience here - please highlight that you are experiencing an issue with HTTP functions. I gave a heads up to the GCF support team that many Firebase users have reported (here and elsewhere) issues with HTTP functions.

Daniel

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-tal...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/4d2b92f9-90f9-494d-bae8-cb78c89f9c8fn%40googlegroups.com.

[Robbert]

Just a quick follow-up: for us, the issue has stopped by itself around September 2. Since September 2, we've had just 1 "request was aborted because there was no available instance" on September 7, so nothing like before. Did something change for the others as well? Was there any fix related to this on Google Cloud?

-Robbert

[Supriya Aggarwal]

We were still seeing the issue until 2 days back but looks like the GCP team ack'ed and resolved the issue yesterday, so we hope we won't see it any longer.

https://status.cloud.google.com/incidents/16SSwVXrYSLjy8fEMvyZ