Google BLOB service stopped to upload files in US region,

186 views
Skip to first unread message

Henry Saldyga

unread,
Feb 19, 2020, 9:53:58 PM2/19/20
to Google App Engine
Have same Java application on  Google application engine in US region  and in Australia region  that using BLOB storage. And functionality used to be working for about 6 years. 

3 days ago started to get an 500 internal error when uploading a file to the BLOB store using the upload URL received from application in US region.
Uploads to the BLOB storage with URLs received from application in AU regions still working.
Looks like something wrong on Google side ?

Aref Amiri (Cloud Platform Support)

unread,
Feb 20, 2020, 2:18:57 PM2/20/20
to Google App Engine
Henry,

There has been an wide spread incident with Blobstore as of Feb 19, 2020 however, this has been resolved by the engineering team. If the issue persists on your end, I'd suggest to create a Private Issue Tracker and provide project name so that it can be investigated further.

Henry Saldyga

unread,
Feb 20, 2020, 2:32:24 PM2/20/20
to Google App Engine
Thanks I have already created an issue there.

Jason Li

unread,
Aug 19, 2020, 4:15:07 AM8/19/20
to Google App Engine
This issue happened again in US region a few days ago and it's still broken. 

David (Cloud Platform Support)

unread,
Aug 19, 2020, 3:02:40 PM8/19/20
to Google App Engine

Jason, if you are still experiencing this issue and you believe it’s an issue on our end, you are able to create a private Issue Tracker and provide information such as Project ID. If determined this issue is not only individually affecting you, an investigation will be launched. Otherwise, you are also able to open a support ticket with GCP support if you have access to a support package.


Henry Saldyga

unread,
Aug 19, 2020, 3:23:32 PM8/19/20
to Google App Engine
Of you can do that Jason, but since February 20th I created 5 private issue trakers but problem still not solved. 

Elliott (Cloud Platform Support)

unread,
Aug 20, 2020, 5:21:17 PM8/20/20
to Google App Engine
Hello Henry,

I'm sorry that you've had to do through this multiple times. I've looked at one of the issue trackers you opened with us. The analyst could not reproduce your scenario and left a message for you.

Henry Saldyga

unread,
Aug 20, 2020, 5:49:30 PM8/20/20
to Google App Engine
Do you want to say the analyst managed to upload an image to the BLOB? 
Please provide an URL to this BLOB please?

Jason Li

unread,
Aug 21, 2020, 4:50:36 AM8/21/20
to Google App Engine
It looks like BlobStore upload is still broken. We use Python 2.7 and here is the boilerplate code used in our system:

class MainHandler(webapp2.RequestHandler):

  def getUploadForm(self):
    upload_url = blobstore.create_upload_url('/imageUpload')

    self.response.write('<html>\n<body>\n')
    self.response.write('<form action="%s" method="POST" enctype="multipart/form-data">\n'% upload_url)
    self.response.write('<input type="file" name="file" accept=".png"><br>\n')
    self.response.write('<input type="submit" name="submit" value="Submit"></form>\n')
    self.response.write('</body>\n</html>\n\n')
  .....

After the user hits the Submit button, a 500 error will be returned. The expected behavior is that the image will be uploaded to BlobStore and then navigate to myDomain.com/imageUpload. Note that the 500 error occurred before navigating to image upload success handler myDomain.com/imageUpload. Because the issue is not on our codebase, we cannot see any errors/traces on Google Cloud Console/App Engine Dashboard.

The same code has worked for at least 5 years.  We understand that we need to move to Google Cloud Storage and switch to Python 3, but we have a complex system and it will be a huge undertaking for us. I really hope Google can fix this issue as soon as possible. Thanks!!
============================
    1. Request URL:
    2. Request Method:
      POST
    3. Status Code:
      500
    4. Remote Address:
      [2607:f8b0:4000:816::2013]:443
    5. Referrer Policy:
      no-referrer-when-downgrade
  1. Response Headers
    1. content-length:
      0
    2. content-type:
      text/html; charset=UTF-8
    3. date:
      Fri, 21 Aug 2020 02:53:14 GMT
    4. server:
      UploadServer
    5. status:
      500
    6. x-guploader-uploadid:
      AAANsUkAqnyXebkSBTKiYHZ4ObP9rk0weOQ1jiBRXpkDQhdc73HeJydQvAIR6_bl-Z-VG-QWfWqDMe4FP3ErHqdYMObvuyA44A
  2. Request Headers
    1. :authority:
    2. :method:
      POST
    3. :path:
      /_ah/upload/AMmfu6axah0bzR6QPuVaEb1I3WnGVpuSAU11QSdFFGLWFl15YMvLUJEqN6lZZw_RxH6pQp7TOORbPEApZe4OZNnreTAjLNiYuKgp0AGlghpIZ_iYtUDLmKSSO11lTEecdu_sFcQV3n7qsPZ28OWss_NVN6c5CA2p794poyCYpBWiLBarvdCksLtKJJA0IUiuNcyJQRD6FmDPHonHngD0d8FImpWqa6_XNA/ALBNUaYAAAAAXz85bB_Pf7F0dGmJhgGZNsyQ9RN_d5lv/
    4. :scheme:
      https
    5. accept:
      text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
    6. accept-encoding:
      gzip, deflate, br
    7. accept-language:
      en-US,en;q=0.9
    8. cache-control:
      max-age=0
    9. content-length:
      205103
    10. content-type:
      multipart/form-data; boundary=----WebKitFormBoundaryyCwtZU7W4GyPos5l
    11. cookie:
      __utmz=157282283.1597952850.1.1.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=(not%20provided); __gads=ID=53591dab302103ee:T=1597952850:S=ALNI_MaF4YdKpIKyFr4pSlgJeMJlFfmocQ; __utmc=157282283; __utma=157282283.1009616170.1597952850.1597965252.1597975354.3; __utmt=1; __utmb=157282283.9.10.1597975354
    12. origin:
    13. referer:
    14. sec-fetch-dest:
      iframe
    15. sec-fetch-mode:
      navigate
    16. sec-fetch-site:
      same-origin
    17. sec-fetch-user:
      ?1
    18. upgrade-insecure-requests:
      1
    19. user-agent:
      Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.135 Safari/537.36

Jason Li

unread,
Aug 21, 2020, 5:08:02 PM8/21/20
to Google App Engine
Is it possible to let us know if this BlobStore upload issue will be fixed?  Thanks!

Mary (Google Cloud Support)

unread,
Aug 26, 2020, 7:12:13 PM8/26/20
to Google App Engine

Hello, 

There was a service interruption with BlobStore on the 21st which was resolved. If you are still experiencing issues, we recommend to open a support ticket via the Support Hub[1] to further investigate into your issue. 


Reply all
Reply to author
Forward
0 new messages