Error when deploying a Firebase cloud function

[Sameer Madan]

Hi! I'm seeing an error when trying to deploy a cloud function.

"Step #2 - "analyzer": [31;1mERROR: [0mfailed to initialize cache: failed to create image cache: accessing cache image "us.gcr.io/MY_PROJECT/gcf/us-central1/SOME_KEY/cache:latest": failed to get OS from config file for image 'us.gcr.io/MY_PROJECT/gcf/us-central1/SOME_KEY/cache:latest'"

I can confirm that I can build and test my cloud function locally. I was able to emulate it locally and it works fine.

Any idea on what's going on? The error message is not very useful.

[Alexis (Google Cloud Support)]

Hi,

It will be a pleasure to help.

It's not clear to me if you are using Cloud Build or not to deploy your Cloud Functions, since I see "step#2". I also think that the words "failed create image cache" mean that it's related to access control for Container Registry (gcr.io).

This[1] shows the default permissions for Cloud Build may have changed since Google favors the new Artifact Registry instead of the older Container Registry. Ensure these roles[2] are used on the proper Cloud Build or Cloud Function service account in order to access Container Registry (gcr.io) properly. If you are using Cloud Build to deploy functions, these[3] additional permissions may also help.

Let us know if that worked.

[1] https://cloud.google.com/build/docs/cloud-build-service-account

[2] https://cloud.google.com/container-registry/docs/access-control#permissions_and_roles

[3] https://cloud.google.com/build/docs/securing-builds/configure-access-for-cloud-build-service-account#granting_a_role_using_the_settings_page

[Sameer Madan]

Hi Alexis,

It's definitely not a permission issue because:

1. I can create a new cloud function and it deploys successfully.

2. I can update any new cloud functions that I've created above.

3. I cannot update any of my existing cloud functions. Every single one fails to update. However, if I comment it out in code, deploying that change effectively deletes the function. After that, I can successfully re-create it. This is not a workable solution for any functions that are already deployed.

I'm writing my cloud functions in Typescript and deploying using the CLI: "firebase deploy --only "functions:myCloudFunction".

[Derrick Chao]

Sameer, did you ever get this working?

Alexis, I am also getting the exact same error this morning (using both the Google Cloud front-end as well as gcloud CLI) and all of my Billing Health is fine. I have been previously deploying these same functions without issue for many months.

Are you able to provide more step by step instructions on how to update this? I have tried to update the permissions for us.artifacts.xxxx and granted 'owners of' and 'editors of' my project the updated Storage Owner Admin permission, but it is still failing. Under Cloud Build permissions, everything is DISABLED like this screenshot: https://cloud.google.com/build/docs/securing-builds/configure-access-for-cloud-build-service-account

I apologize, I am not well-versed in IAM and don't want to mess anything up. 

[Derrick Chao]

Alexis, this is a bug given the reasons Sameer has stated. This is not limited to Node.js runtime, I also tried to update a Java runtime function and get the same error. As Sameer mentioned, creating a new function works, but deploying any existing function fails. As I have multiple Cloud Functions across multiple apps, performing this workaround is a huge pain.

While creating a new function works, copying a Cloud Function and trying to deploy that also fails with the following attached error message which might hopefully be instructive to the Google engineers.

In my case, I have a LIFECYCLE rule set up for 3 days to automatically delete old storage bucket artifacts, not sure if that's related.

[Sameer Madan]

If it helps, I too have a lifecycle rule set for 1 day to automatically delete old storage bucket artifacts.

[Sameer Madan]

Yeah I'm pretty sure this is happening due to the lifecycle rule - I deployed a new function on Friday and today I can't re-deploy it. The only thing that has happened over the weekend is that the lifecycle rule removed artifacts. This seems wrong - that artifacts bucket uses a LOT of storage and I really don't want to pay for it to store old images of my functions. As such, the lifecycle rule is very important to keep. Can someone at Google please look at this and resolve?

On Monday, May 3, 2021 at 5:33:04 PM UTC-4 derrick...@gmail.com wrote:

[Felipe Bergallo Corral]

Hmmm...

As I understand it currently, the caching is part of the process, it utilizes the previous build as a jumping off point to reduce build time, which is why a new one can work with the same code; however the case that Derrick R. Chao is showing seems to be different from yours, Sameer, given that you've not mentioned a permissions error in any of your messages.

So what I would like to know is if the image Derrick has attached corresponds with the error that you're getting, Sameer?
And if it is, have either of you found any success in granting the user the "actAs" permission as the error instructs you to?

[Derrick Chao]

While I'm not sure if the issues are related, I am getting all of the same errors as Sameer and I received the permission error only when trying to copy an existing function and then deploy it. I imagine Sameer might as well if he does the same. 

We have an existing issuetracker open for this as others are encountering the same issue: https://issuetracker.google.com/issues/186832976

[Felipe Bergallo Corral]

Very well, if Sameer can confirm that their issue is one and the same, it would be best to check up over there, rather than here, that way it's easier to keep track of. Otherwise I will likely ask Sammer some more questions if they're available.

[Sameer Madan]

I don't believe I've seen any permission error. In my case, the error I saw is the one I noted in this thread in my first post. The only workaround for me was to delete and re-create that function. However, even in that case this newly 
created function seems to cause a problem when I try to update it after a couple of days. I just tried doing that with a function I created 2 days ago and I got this error:

"Step #5 - "exporter": [31;1mERROR: [0mfailed to export: failed to write image to the following tags: [us.gcr.io/MY_PROJECT/gcf/us-central1/SOME_KEY:myCloudFunction_version-3: GET https://storage.googleapis.com/us.artifacts.gezunt-c3bfe.appspot.com/containers/images/sha256:SOME_HASH?access_token=REDACTED: unexpected status code 404 Not Found: <?xml version='1.0' encoding='UTF-8'?><Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Details>No such object: us.artifacts.gezunt-c3bfe.appspot.com/containers/images/sha256:SOME_HASH</Details></Error>]"

I have a lifecycle rule in place that empties out artifacts after 1 day. I strongly suspect that might be the issue here, but I'd love to hear your thoughts on how I could keep that rule in place but still be able to deploy and update functions.

[Felipe Bergallo Corral]

Keeping that rule in place will almost certainly keep causing this issue for you as there are parts of the image layer that are stored there and your rule is deleting them. For a comparison it would be as if you deleted part or even the entirety of the "System32" folder in Windows to make space. You'd have to reconfigure the rule to avoid the pattern associated with images. Given that you've stated that you're using this lifecycle rule to empty out artifacts and not just some artifacts, I don't believe you can keep this rule in place.

[Sameer Madan]

I see. The reason I put that rule there is due to this: https://stackoverflow.com/questions/63843721/firebase-storage-artifacts-is-huge-and-keeps-increasing.

Basically I saw almost 1GB of storage being used by images created from my functions which are a grand total of 500 lines of code. I want to figure out a way to not have to pay for all this unnecessary storage and that post I just linked was the only solution I found. Do you have an alternative that I could look into?

[Felipe Bergallo Corral]

That's the thing... It's not unnecessary, while you might consider the subsequent images superfluous because they're incremental, the initial image created when you deploy is not unnecessary. It contains the information that Cloud Functions would need to deploy your code on an instance to then run that code unimpeded, as when you are not calling said function, the instance that runs the code is deleted, and all that's left is the image.
To try and explain myself a little better, I'll try and put the process as a bulletpoint list.
The first time you deploy a function the process is essentially this:
 1) Grab existing base image for the appropriate language
 2) Use said image to create an instance
 3) Apply user code to the instance
 4) Save said changes to an image ; create bucket if it does not exist
 5) Deploy image with user code to instance; destroy (instance's container) when no longer use
Every time thereafter, the process would go a little something like this:
 1) Check for bucket --> If bucket doesn't exist, create it, and go through first-time process
 2) If the bucket exists, grab the image --> The image is expected here, if it is not found, it throws an error. At minimum, the initial or latest version should be here
 3) Deploy image with usercode to instance

As you might've noticed, every time after the first, the deployment is about half the length as the initial one, if not more (at least in steps).
As for the pricing... Lets start by looking at storage:
https://cloud.google.com/artifact-registry/pricing#storage

1GB of storage is... .10$ That is to say 1/10th of a dollar,  a little over 8.3 cents of a euro (83/1000ths), 7.36 indr, almost 11 yen (10.9); and that would be every month.
Now, I can perfectly understand that, as much as that may be nothing where I live, it can be expensive or at the very least noticeable to someone else depending on their salary, the cost of living, etc. However, given that functions only have multiple images if you are generating new versions, and it's entirely possible that you may need to revert to a previous version to rollback some unfortunate changes, I would not call this unnecessary storage.

What's more, the stackoverflow link you've sent includes warnings of this exact error (and I'm not sure why one would upvote this kind of workaround when people are having issues with it...) :
"

NOTE: if you face any deployment issues later, like if you deploy several days in a row and if it gives you an error on deploy, just delete the whole "container" folder manually in the artifacts which should solve it and then redeploy again. (make sure not to delete the artifacts bucket itself!)

Hope the firebase team will improve that - the current behavior looks confusing as it easily leads to an unexpected bill unless you take extra steps to prevent that. But you'll never know that it will happen until it does.

"
And while I disagree on the current behaviour "looking confusing", I can understand that our documentation can seem misleading... But anything we obviate is because it's part of the documentation, it's not something we hide or attempt to confuse anyone about. There's clear notifications on a Function's version, and you have the ability to go back to previous versions if you so desire. In fact, removing old versions of functions is quite easy so long as you modify old versions of functions like so:
https://firebase.google.com/docs/functions/manage-functions#modify
And rely on implicit deletion (mentioned here: https://firebase.google.com/docs/functions/manage-functions#delete_functions ) or, alternatively, delete them manually.

PS: This https://gist.github.com/kichiemon/4ba5bf921bc9e4d208db8723da69f0ed was recommended on https://issuetracker.google.com/issues/186832976#comment18 ... And the descriptions don't seem to be related... From what I've read of that thread (which is all of it), everyone on it is having different errors, and... Just, don't do this, especially not if you're deleting the images that instances use to run your Functions. I trust the Firebase Team, and I trust that they did such a thing as a last resort, but it has little to do with your case, as it seems that particular commenter's containers were stuck in some way. 

[Sameer Madan]

Thanks for the detailed response! Sounds like I'll just remove the lifecycle rule for now. However, I would like to point out a couple of things:

- However, given that functions only have multiple images if you are generating new versions, and it's entirely possible that you may need to revert to a previous version to rollback some unfortunate changes, I would not call this unnecessary storage.

I suppose the exact dollar amount for storage is indeed not high (I live in the US) and I can live with paying a few cents a month. However, I'd argue that there should be a way for users to specify whether they want rollback capabilities or not. This is a non-trivial sized storage footprint that I imagine most people don't need. I don't recall seeing this type of storage-heavy rollback capability for server code even during my time working at a big tech company. I would much rather simply rely on my own source control to revert code back to an older version (if I need to do so) and simply deploy that change.

- But anything we obviate is because it's part of the documentation, it's not something we hide or attempt to confuse anyone about. There's clear notifications on a Function's version, and you have the ability to go back to previous versions if you so desire.

I'm definitely not accusing you all for making this confusing on purpose :). My point is that in its current form, cloud functions are optimizing for rollback capability in a way that is non-intuitive. From the point of view of designing a good piece of software, I would put this feature as something that's defaulted to off, but can be turned on if the user chooses to use it. At that point, the storage implications should be made clear. I was definitely surprised to see storage utilization and had to look for answers as to what was using all this space. I would almost flag this right in the instructions that tell you how to deploy a function: https://firebase.google.com/docs/functions/manage-functions#deploy_functions. Right now, it does not mention storage usage at all.

Thanks for looking into this!

[Felipe Bergallo Corral]

Gotcha, ok, so:
Regarding the first point, these are incremental backups, it's not a full version. Still, I do agree that I believe it should be an option to just have the one version.
Regarding the second point, this is probably my lack of soft-skills, but what I meant was: if there's any part of the documentation that uses odd wording or that implies that the behaviour is not what you'd expect, my recommendation would be to copy portions of the document and google them (generally adding "gcp" between quotes after leads to our documentation), because it's not totally unheard of that we establish a set of words to be referencing something specific but then don't link back or mention that establishment earlier. If you find anything like this, you can report, and we would encourage you to, given that our documentation can be unintuitive in some languages or due to odd punctuation.

Going back to the first point: have you found or created a Feature Request for this? I've looked for it myself but haven't found it. If it exists, I'd say comment on it to promote it; and also link it, for anyone who finds this conversation.

[Felipe Bergallo Corral]

* "given that our documentation can be unintuitive in some languages or due to odd punctuation." Odd punctuation, or even odd phrasing. Sorry.