Blobstore API upload behavior on error

[Jeff Schnitzer]

What is the behavior of the blobstore upload API when the callback produces a 500 error? This is not at present documented.

I would hope that GAE deletes the blob, otherwise it's going to leak. My application cannot be trusted to detect the 500 error and delete the blob explicitly; after all, the 500 could have been caused by a timeout or something else out of my application's execution control.

FWIW, I'm specifically interested in the behavior of the upload API when backed by GCS, although the same question applies to traditional Blobstore.

Thanks,

Jeff

[Patrice (Cloud Platform Support)]

Hi Jeff,

It really depends on where exactly your 500 is returned. If it's in the callback, I believe that will mean your blob was properly uploaded.

In your situation, when my application receives a 500, I would check for timeouts to be certain this isn't the issue, then I'd check for file existence, and delete as needed.

Cheers

[Jeff Schnitzer]

I don't quite understand. What happens in this circumstance?

1) My application returns the result of createUploadUrl() to the client (browser).

2) Browser uploads file to that URL.

3) Google tries to call my callback and times out

#3 could be my fault or it could be GAE's; either way, from the perspective of my code, I don't know anything about whether the upload completed. Did this just orphan some stored data in GCS?

Jeff

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/bab09cc1-467d-42c9-898f-49085e7887be%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Patrice (Cloud Platform Support)]

Hey Jeff,

Thank you for coming back so quick. Yeah, reading my own reply, I'm not clear on it either..... so I guess I needed more coffee before I wrote that.

Anyway, basically, in that specific circumstance, if Google tries to call your callback, then it means the url was successful in uploading and, indeed, you will have a blob added. Then yes, in that situation, your blob entry would be orphaned and would need to be cleaned out.

If you receive the 500 during step 3, you most likely have an orphaned blob. If you receive it during step 2, chances are your blob never got fully uploaded.

I hope this clarifies it. Lemme know if it doesn't

Cheers

On Tuesday, June 2, 2015 at 1:47:28 PM UTC-4, Jeff Schnitzer wrote:

I don't quite understand. What happens in this circumstance?

1) My application returns the result of createUploadUrl() to the client (browser).

2) Browser uploads file to that URL.

3) Google tries to call my callback and times out

#3 could be my fault or it could be GAE's; either way, from the perspective of my code, I don't know anything about whether the upload completed. Did this just orphan some stored data in GCS?

Jeff

On Tue, Jun 2, 2015 at 6:59 AM, Patrice (Cloud Platform Support) <pvout...@google.com> wrote:

Hi Jeff,

It really depends on where exactly your 500 is returned. If it's in the callback, I believe that will mean your blob was properly uploaded.

In your situation, when my application receives a 500, I would check for timeouts to be certain this isn't the issue, then I'd check for file existence, and delete as needed.

Cheers

On Monday, June 1, 2015 at 9:59:53 AM UTC-4, Jeff Schnitzer wrote:

What is the behavior of the blobstore upload API when the callback produces a 500 error? This is not at present documented.

I would hope that GAE deletes the blob, otherwise it's going to leak. My application cannot be trusted to detect the 500 error and delete the blob explicitly; after all, the 500 could have been caused by a timeout or something else out of my application's execution control.

FWIW, I'm specifically interested in the behavior of the upload API when backed by GCS, although the same question applies to traditional Blobstore.

Thanks,

Jeff

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.

[Patrice (Cloud Platform Support)]

Hi Jeff (and whoever else follows this thread).

Just as a headsup, I got a private email sent by Jeff saying that he opened a public issue tracker to get this looked up and changed.  I (or another member of the team) will be happy to look it and send it to the proper team.

Cheers!

[Jeff Schnitzer]

Oops! That wasn't meant to be private, thanks for letting me know. For anyone listening, here's my reply and a link to the story (for starring purposes):

----

Thanks for the quick response!

This seems like pretty undesirable behavior. The only way to cull orphans is to write a batch process that walks through my whole GCS bucket (potentially millions or billions of entries) checking them one at a time. Yuck. It would be much better to prevent orphans from happening in the first place.

I opened this defect report: https://code.google.com/p/googleappengine/issues/detail?id=12015

Thanks,

Jeff

Cheers!

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.