Uploads to Legacy Blobstore are Failing with 500, nothing in the logs

[Joshua Smith]

There is a production issue in my app. (I’m happy to create a private production issue ticket, but I couldn’t figure out where to do that.)

My app is still on Python 2.7 and uses the legacy blobstore.

I use blobstore.create_upload_url('/upload’) to populate a form. This has been working fine for several years.

The last successful one was: 02/Oct/2020:11:37:23 -0400

As of this morning, attempts to upload to the blobstore are resulting in error 500 and I don’t see the request in my logs.

I simulated one of these uploads using curl, but that produced no insight:

* Using HTTP2, server supports multi-use

* Connection state changed (HTTP/2 confirmed)

* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0

* Using Stream ID: 1 (easy handle 0x7fa064808200)

> POST /_ah/upload/AMmfu6YC1hHSvQPFczp—redacted—Pt3zS4LmedW5GR/ HTTP/2

> Host: —redacted— 

> User-Agent: curl/7.64.1

> Accept: */*

> Content-Length: 12881

> Content-Type: multipart/form-data; boundary=------------------------c1bce33de7c355c9

> 

* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!

* We are completely uploaded and fine

< HTTP/2 500 

< x-guploader-uploadid: ABg5-UyVR-AC3_kY-A3vD45EgWUWJjzynvU_S0rDjhhsIlyJsDmYRU-ZrjSLe9Pppe7eCd-S9dHAv_7dVcsvfMj5wS0

< content-length: 0

< date: Mon, 05 Oct 2020 15:38:41 GMT

< server: UploadServer

< content-type: text/html; charset=UTF-8

< 

* Connection #0 to host mytowngovernment.org left intact

* Closing connection 0

Although perhaps that uploadid might help you diagnose?

Please advise.

-Joshua

[Joshua Smith]

The silence here is deafening, but my users are howling.

Google: Please address this issue immediately! It’s been down at least 24 hours!

I created an issue in public issue tracker with the same info you see here: 170195261

I highly doubt I’m the only one experiencing this.

-Joshua

[Olu]

Hi, Joshua

Thank you for reporting this issue. It seems you may be affected by an internal issue reported about App Engine standard applications returning elevated HTTP 500 errors. This issue is observed for App Engine  Standard instances in us-central1 region. This issue is externalized on the GCP Status Page as you can find more information on the Status board[1].

I apologize for the delay or the impact this issue may be causing. For swifter responses or help on issues that are urgent to you, I think it is best to reach out directly to the GCP Support Engineers[2]. 

Let us know if your application is not running in the us-central1 region.  

Thank you. 

[1]https://status.cloud.google.com/incident/appengine/20007

[2]https://cloud.google.com/support-hub

[Joshua Smith]

It pre-dated that issue, and it is continuing despite the larger outage being resolved.

Please address this ASAP.

Thanks.

-Joshua

-- 
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/9e37a2e5-ef32-481c-87f2-fff0f807c5dan%40googlegroups.com.

[Joshua Smith]

On Oct 6, 2020, at 11:47 AM, 'Olu' via Google App Engine <google-a...@googlegroups.com> wrote:

I apologize for the delay or the impact this issue may be causing. For swifter responses or help on issues that are urgent to you, I think it is best to reach out directly to the GCP Support Engineers[2]. 

My reading of that the cases page is that I’d have to pay you $100/month for the privilege of reporting outages that way. (The free tier says it’s read-only.)

Am I missing something?

-Joshua

[Joshua Smith]

The problem persists.

It occurs to me that last week was the first time I had to make a small update to the app in a while, and I did that using “gcloud app deploy”  instead of appcfg.py for the first time on that project.

Is it possible that pushing the app using gcloud introduced a problem uploading to blobstore? I wouldn’t think they’d be related, but the timing is suspicious.

Downloading from blobstore still works. It’s only uploads that are failing.

My users are getting upset, and your lack of feedback or an ETA is not helping.

-Joshua

On Oct 6, 2020, at 11:47 AM, 'Olu' via Google App Engine <google-a...@googlegroups.com> wrote:

[Joshua Smith]

Sigh.

I’ve gone ahead and re-coded my app to use cloudstorage instead of blobstore for new uploads. So this is no longer critical.

(Thankfully, downloads have continued to work. If you break blobstore downloads, that’ll be a real problem for me.)

The idea of using a cloud provider is they:

1. Keep the infrastructure working, and

2. Respond quickly when there are outages.

If I have to spend $100/month to get that level of service, it changes the economics completely.

I’m very disappointed in GCS right now.

-Joshua

[Charounson Saintilus]

Hey Joshua,

I don’t think I’m gonna be of much help here, but wanted to mention: if you think that running “gcloud app deploy” introduced the issue, can you try redeploying the application using appcfg.py? 

You could also use curl commands to troubleshoot the upload to blobstore.. 

Thanks,

Char 

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/9EC69CDD-FDB4-4CEA-AFA2-BC03919C300A%40gmail.com.

[Joshua Smith]

Unfortunately, I used gcloud app deploy because appcfg.py stopped working a couple weeks ago.

I did use curl and it revealed nothing. Just a terse 500 with no content from the server.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CAHRfKp6Osqzm_ca55Ck2isvaxBAw4VN4QViAcoVE5FwqaQKhSg%40mail.gmail.com.

[barrado]

Hi,

I have deployed this Blobstore API sample service and it works well and allows me to successfully upload images. You could try to add further logging statements in your app to know what is causing your issue. 

On the other hand, I recommend you to fully migrate you app to Python3 since Python 2 is no longer supported by the community and use Cloud Storage instead of Blobstore.

And also note that appcfg tooling was shuted down on August 30th, 2020.

[Will Reiher]

I had an issue a few weeks back. It was due to my transition to gcloud deploy. It created a new app-id.appspot.com bucket but my app engine service account did not have permissions. Once I gave it permission it started working again with the blobstore items showing up in the new "default" bucket.

[Joshua Smith]

Do you recall if you got 500 errors? I’d expect that to generate 403’s.

Regardless, that does sound like the kind of thing that happened.

Switching to use cloudstorage instead of blobstore wasn’t too hard, so that’s the workaround I’d recommend for anyone else who hits this issue.

I’m mostly just annoyed that google has switched to a support model where they won’t provide timely support unless you pay them $100/month. That’s greedy and stupid.

-Joshua

How we’re addressing COVID-19
Health and safety is our top priority. We have made arrangements for all staff to work remotely from home to continue to provide uninterrupted service, expertise and advice to our authors and prospective customers during the unfolding COVID-19 situation. Click here for more information.

Connect with us at:

FriesenPress Mailing Address | Suite 300 - 990 Fort St, Victoria, BC, Canada, V8V 3K2
friesenpress.com | friesenpress.com/blog

You are receiving this email because you are a FriesenPress client or partner. This email and any documents attached to it may contain confidential information. If you are not the intended recipient, you are hereby notified that any disclosure, copying or distribution of the contents of this email is prohibited. If you no longer wish to receive promotional emails from FriesenPress, please click here to change your settings.

The privacy of your information is important to us. To learn more, visit our privacy policy.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/842f48d2-243d-447b-8d47-2a0fb7c9c691n%40googlegroups.com.

[Will Reiher]

I don't remember what the error code was anymore. I just knew that something was f'd. It was really a hunch that that new bucket was involved and that if my app didn't have permission it might be the answer. I was prepared for a very long day but after the permission change it all just started working again - with the blobs showing up in that new bucket.

Most of my stuff is using Cloud Storage except for user uploads. I'm working to replace that with a UUID based storage name so that I can maintain the ability to allow for same-named files to be uploaded.

Of course today Cloud Task UI is not available and it wants me to enable the API (I've been using it for months now) Nervous that my queues will be erased if I enable it.

[Jose V]

Hello,

I have seen previous occurrences in which deploying with "gcloud app deploy" led to 50X errors when uploading to blobstore. It was indeed due to the App Engine Service account lacking permissions on the newly created Cloud Storage bucket.

Note that by default, the Service account used by App Engine has a name like "[PROJECT-ID]@appspot.gserviceaccount.com" and the Cloud Storage Bucket will have a name like "[PROJECT-ID].appspot.com"

Could you confirm that the App Engine service account has write permissions for the bucket? See Cloud Storage Roles [1] for reference.

Regards
[1]: https://cloud.google.com/storage/docs/access-control/iam-roles

[Joshua Smith]

Indeed, there is no such account with any permission on that bucket.

So there you have it. That’s probably what happened. (Doesn’t matter anymore for me, since I’m now using cloudstorage for uploads, not blobstore.)

Google really botched this transition from appcfg to gcloud.

-Joshua

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/880f9f5c-f28c-4b0f-9cf6-5827235aa128n%40googlegroups.com.