Outage?

[Joshua Smith]

I’m trying to update a website deployed in appengine. I’ve made sure my cloud SDK is all up to date. It’s a big site, with about 8000 files. I’ve been using the same deploy command in this same folder for years. I’m getting this cryptic error:

Updating service [default]...failed.                                                               

ERROR: (gcloud.app.deploy) Error Response: [5] failed to fetch metadata from GCR, with reason: generic::not_found: failed to fetch metadata from GCR for image us.gcr.io/kaoncom-hr/app-engine-tmp/app/ttl-2h:8927ab89-5aed-4a70-9969-9ed1927844ec, with reason: generic::not_found: fetchImageMetadata failed for image us.gcr.io/kaoncom-hr/app-engine-tmp/app/ttl-2h:8927ab89-5aed-4a70-9969-9ed1927844ec, reason: generic::not_found: failed to fetch manifest from GCR (via gcr.FetchManifest): generic::not_found: error fetching "kaoncom-hr/app-engine-tmp/app/ttl-2h/manifests/8927ab89-5aed-4a70-9969-9ed1927844ec" : generic::not_found: got HTTP/404 response, wanted HTTP/200

The guid-looking thing changes every deploy. kaoncom-hr is the name of the app.

I checked the dashboards and google is not reporting an outages.

Any ideas?

-Joshua

[Joshua Smith]

Digging into the GAE “Activity” logs, I see some new stuff about google cloud builder that I haven’t seen before.

Perhaps something went sideways in the transition to that new technology?

All these extra service accounts and whatnot that got created and show up in IAM are ridiculously complicated. I wouldn’t know where to begin diagnosing that.

What’s the right way to get someone from google to figure out how they messed up my account?

-Joshua

[Joshua Smith]

So this problem persists, which I guess means it’s not an outage, but rather something got messed up by google on my account.

Please advise on how I can get this issue rectified. To summarize, I’m running gcloud app deploy and it is reporting a 404 error trying to read the manifest of the cloud bucket where it is staging things.

I looked at the issue tracker, but that doesn’t appear to be geared toward production problems any more.

Opening a ticket appears to have a $100/month price tag attached to it, which seems onerous considering how much my company is paying google every month for these services, and considering google broke my account.

What’s the right course of action here?

-Joshua

[joshuamo]

App Engine will be moving towards using Cloud Build for the Standard environment[1], matching the other Serverless offerings.  Good catch noticing the Cloud Build aspect here.  We were rolling this out at a low percentage and this is what seems to have triggered the issue you've experienced.

For now, we've rolled back the experiment, so you should be able to deploy again without issue.

[1] https://cloud.google.com/appengine/docs/standard/payment-instrument

[Joshua Smith]

Confirmed that my deploy is now working. A little heads-up would be nice before you take another shot at that experiment. I don’t remember opting-in to being a guinea pig.

And yes, I assumed it had something to do with that cloud build stuff. I guess I understand why you are doing it, but this transformation of Google App Engine from a PaaS offering to an IaaS offering is horrible. It used to be we had a simple Launcher app that made development and deployment of an app super easy. Now I’m dealing with a half dozen “services,” trying to ensure all the service accounts have the right permissions, and having to learn entire stacks to do something that used to be an API call.

There is a real market need for a PaaS offering, which is what GAE used to be. Now it’s gotten so complicated that I might as well be spinning up a container in EC2. You have completely destroyed your competitive advantage, and there is vacuum that someone (beanstalk maybe?) is going to step into.

It reminds me of the the way Microsoft Access solved a real problem for millions of users, yet Microsoft repeatedly tried to kill the thing to drag people to SQL Server, which was such overkill for 99% of those users.

Once again, Google is listening to their engineers instead of their customers. It’s a shame. GAE was great. This gcloud mess is the opposite of great.

-Joshua

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/76bda25a-fc17-4766-a956-ceecf819fc7b%40googlegroups.com.

[Linus Larsen]

Totally agree, for small companies or startups with limited resources we only want our services to run with minimal effort.

To unsubscribe from this group and stop receiving emails from it, send an email to google-a...@googlegroups.com.

[Patrice B]

So I'm not the only one feeling AppEngine is gradually moving away from Paas, although this trend does not seem to be acknowledged officially.

I had recently posted here:  the end of Paas ?

Indeed it seems more and more of the services that made App Engine a proper PaaS offering are being migrated to IaaS.