Broke GCS bucket by deploying a container to an older classic appengine project?

[Ehsan Kia]

3 days ago, I went to deploy a newer AppEngine project using `gcloud app deploy` but forgot that the default project was set to an older project.

I have since reverted that by redeploying the older project, but my Google Cloud Storage seems to be completely broken.

It does not let me access any of the files, either from the UI or gsutil. It throws "Internal server error", and on gsutil it gives me "The App Engine application does not have enough quota".

i see 429 when trying to access from the API and 403 in other places. Trying to transfer the files or download the files or anything just continuously throws errors.

I don't actually see any quotas that are full, and it doesn't matter anyways because I have billing enabled and nowhere close my budget limit. Not sure what "quota" this is referring to.

Nothing seems to fix it, the whole storage bucket seems to be locked up and erroring. Anyone have any idea how I can recover this?

[Ehsan Kia]

For what it's worth, it got fixed the next day as the quotas reset, even though no quota showed as past it's limit and even though I had billing enabled. So if you run into the same issue, just deploy the right thing again and wait a day.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/ba0c56bb-d7fa-4739-b83d-313a20ba5209o%40googlegroups.com.

[Ehsan Kia]

Nvm, this is still happening. The closest I can find is this:

https://stackoverflow.com/questions/61052413/google-cloud-storage-quota-hit-how

But I'm not anywhere near my limit.

[David (Cloud Platform Support)]

Hello,

This error can show up due to various matters usually related to billing or a quota being exceeded. I would recommend you to contact GCP support as they would be able to inspect your project and tell you the reason you are seeing this error message.

[Ehsan Kia]

GCP support was not able to help me. I eventually managed to find the issue by using Audit Logs to view all GCS accesses, and turns out one of my apps was misbehaving and trying to read a file every second.

It still isn't clear to me why:

1. It gave me quota limit errors instead of just charging my billing account

2. Why the limit being hit wasn't shown on any page

3. why it was failing in some strange non-descriptive "internal error" way

It seems like it was actually hitting some internal "rate limit" that not surfaced at all in GCP. Which is strange for a Cloud which is meant to be able to scale up without limits.

[David (Cloud Platform Support)]

I believe you may have ended in the wrong support channel. Basic billing GCP support is provided to everyone. For questions like these that require project’s inspection to investigate an issue about a specific product, in this case App Engine and/or GCS, even though a billing issue may be a possible case, it needs to be assessed by a technical support agent.  

Looking at your questions, note that App Engine has its own quota page and also there’s the main quota page that may have other related quotas as well not displayed within the App Engine quota page but that are being used by your application. Not entirely sure if this was a GAE quota, could have been a GCS quota as well. I do agree that if a quota was being exceeded, it should have been displayed either in the App Engine quota page or the main quota page, and that “internal” is not descriptive enough to know what was happening. Usually, internal error comes with an error code as well, but I can’t confirm or not that this is the case with what you were experiencing.

If you or anyone else ever experiences something like this again, I would recommend you to try to check if there’s an error code first and google it. If there’s no information about it, then it would be a good idea to contact GCP technical support in order to get a speedy and accurate response. However, you would need to sign up for a support plan or if you are in a free trial, you are able to get free trial technical support as well. Billing support would not be able to provide any help for these types of issues. 

Additionally, if you ever notice an issue with any GCP product, and you believe it’s a general issue on Google’s end and not related to your specific configuration, you are able to create an Issue Tracker while the issue is reproducible and if determined that the issue qualifies as a product bug, an investigation would be launched. Please note that Issue Tracker is meant to report general issues and feature requests related to products within the GCP suite.

[Ehsan Kia]

> it needs to be assessed by a technical support agent.  

Yes, that's basically the conclusion we came to, that it was beyond billing, and that it wasn't included in free support, so that was a dead end for me at least.

> Not entirely sure if this was a GAE quota, could have been a GCS quota as well.

I definitely checked multiple times all the existing quota pages, dug into them with various filters and ordering for a good hour. It definitely was not something that shows up on the quota page. In hindsight, it seems more like a "rate limit" than a "quota limit" actually, and the HTTP response returned was 429 (Too Many Requests), which is different from the usual "Quota Exceeded" response. That being said, it reset exactly at the same time as the quota would, at midnight PST.

> I would recommend you to try to check if there’s an error code first and google it.

I did extensive search. The closest results was: https://stackoverflow.com/questions/61052413/google-cloud-storage-quota-hit-how

That matches the error I was seeing, but the resolution there was billing limit. Mine definitely was not.

After a bit more search, I think this may have been the issue:

https://cloud.google.com/storage/docs/request-rate#ramp-up

Although I have a hard time believing I was above 5000qps read, and it still doesn't explain why it was broken for over a week, nor why nowhere in the UI I was told what the problem was.

Thank you for the Issue Tracker link though, that will come in handy for searching existing issues and posting new ones.