Blobstore API Feedback
Piotr Sikora
lately I've been playing around with Blobstore and I believe that it misses
few critical features:
1. MD5/SHA1 integrity checksums. This is a *MUST*. All competing services
(S3, Nirvanix, etc) provide ways to:
- get checksum of already uploaded file (this should be another BlobInfo
property),
- upload file along with its checksum. When calculated checksum of uploaded
file differs from the provided one, file is discarded and error response is
sent to client.
2. Ranged requests. With large files this is another *MUST HAVE* feature and
hopefully requires no explanation.
3. When "success_path" generates error response after upload (either 4xx or
5xx) with custom error page, it results in "500 Internal Server Error"
response with Google's default error page, because Blobstore requires 3xx
response. In my opionion this is bad, because one cannot return custom error
page. Maybe only 2xx responses should be handled as incorrect?
4. "500 Internal Server Error" response is returned when wrong / old /
not-existing BlobKey is passed via blobstore.BLOB_KEY_HEADER, in my opinion
this should result in "404 Not Found" response.
Also, few times I received "500 Internal Server Error" response when
uploading file(s) to path previously generated using
blobstore.create_upload_path(). Those errors didn't result in *any* entry in
application logs and I wasn't able to find proper way to reproduce this, but
my guess is that this results from either:
- too old upload token (>24h),
- upload token generated for older deployment.
Best regards,
Piotr Sikora < piotr....@frickle.com >
Nick Johnson (Google)
Hello,
lately I've been playing around with Blobstore and I believe that it misses few critical features:
1. MD5/SHA1 integrity checksums. This is a *MUST*. All competing services (S3, Nirvanix, etc) provide ways to:
- get checksum of already uploaded file (this should be another BlobInfo property),
- upload file along with its checksum. When calculated checksum of uploaded file differs from the provided one, file is discarded and error response is sent to client.
2. Ranged requests. With large files this is another *MUST HAVE* feature and hopefully requires no explanation.
3. When "success_path" generates error response after upload (either 4xx or 5xx) with custom error page, it results in "500 Internal Server Error" response with Google's default error page, because Blobstore requires 3xx response. In my opionion this is bad, because one cannot return custom error page. Maybe only 2xx responses should be handled as incorrect?
4. "500 Internal Server Error" response is returned when wrong / old / not-existing BlobKey is passed via blobstore.BLOB_KEY_HEADER, in my opinion this should result in "404 Not Found" response.
Also, few times I received "500 Internal Server Error" response when uploading file(s) to path previously generated using blobstore.create_upload_path(). Those errors didn't result in *any* entry in application logs and I wasn't able to find proper way to reproduce this, but my guess is that this results from either:
- too old upload token (>24h),
- upload token generated for older deployment.
Best regards,
Piotr Sikora < piotr....@frickle.com >
--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
--
Nick Johnson, Developer Programs Engineer, App Engine
Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number: 368047
Niklasro.appspot.com
2010/1/17 Piotr Sikora <piotr....@frickle.com>:
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengi...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>
>
>
>
Here delete in this case
(https://appengine.google.com/blobstore/explorer/delete.do)
which lookes like a struts (java) uri returns
Server Error
A server error has occurred.
And blob is kept. So trying from scratch I by some means should get
backends according to spec.
Piotr Sikora
thanks for reply, but I must disagree ;)
> 500 is the correct response to return in this case, as the URL the user
> requested (your handler script) was found, while the blobkey returned by
> your script was not valid.
You assume that there is some URL-to-blob_key mapping done, which is not
always the case.
For example, I made my Blobstore data available at "/blob/<blob_key>". At
the moment I verify that <blob_key> really exists by getting its BlobInfo
and I return 404 when it doesn't, but in my opinion, this should be
required.
Also, since this is distributed system, race condidion is always possible,
because concurrent request can delete <blob_key> from the Blobstore at any
time, which means that in such case:
REQ1: client: GET /blob/valid_blob_key
REQ1: handler: b = BlobInfo.get(valid_blob_key)
REQ1: handler: b exists
REQ2: client: GET /delete/valid_blob_key
REQ2: handler: BlobInfo.delete(valid_blob_key)
REQ2: handler: return 200 OK
REQ2: client: 200 OK received
REQ1: handler: set_header(BLOB_KEY_HEADER, b.key())
REQ1: handler: return 200 OK
REQ1: client: 500 Internal Server Error received
client will receive 500 Internal Server Error with Google's default error
page (not even ours).
> Error codes should make sense from the client's
> point of view, not the server's.
Yes, and that's exactly what I mean. From client's perspective data at
"/url" was not found.
Also, please note, that from all the things I wrote in my original e-mail,
this is the smallest issue ;)
Nick Johnson (Google)
Hi Nick,
thanks for reply, but I must disagree ;)You assume that there is some URL-to-blob_key mapping done, which is not always the case.
500 is the correct response to return in this case, as the URL the user
requested (your handler script) was found, while the blobkey returned by
your script was not valid.
For example, I made my Blobstore data available at "/blob/<blob_key>". At the moment I verify that <blob_key> really exists by getting its BlobInfo and I return 404 when it doesn't, but in my opinion, this should be required.
Also, since this is distributed system, race condidion is always possible, because concurrent request can delete <blob_key> from the Blobstore at any time, which means that in such case:
REQ1: client: GET /blob/valid_blob_key
REQ1: handler: b = BlobInfo.get(valid_blob_key)
REQ1: handler: b exists
REQ2: client: GET /delete/valid_blob_key
REQ2: handler: BlobInfo.delete(valid_blob_key)
REQ2: handler: return 200 OK
REQ2: client: 200 OK received
REQ1: handler: set_header(BLOB_KEY_HEADER, b.key())
REQ1: handler: return 200 OK
REQ1: client: 500 Internal Server Error received
client will receive 500 Internal Server Error with Google's default error page (not even ours).Yes, and that's exactly what I mean. From client's perspective data at "/url" was not found.
Error codes should make sense from the client's
point of view, not the server's.
Also, please note, that from all the things I wrote in my original e-mail, this is the smallest issue ;)
--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Piotr Sikora
> I don't assume any such thing. Your code asking the infrastructure to send
> the user a non-existent blob is an issue with your code, not with the
> user's
> request, and as such, it should return a 500 status code, not a 404.
did you read the race condition case? It can barely be issue with
application code.
Nick Johnson (Google)
Hi Nick,did you read the race condition case? It can barely be issue with application code.
I don't assume any such thing. Your code asking the infrastructure to send
the user a non-existent blob is an issue with your code, not with the user's
request, and as such, it should return a 500 status code, not a 404.
--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Piotr Sikora
> I did, and it's unfortunate - feel free to file a bug - but it's still far
> from clear that this should result in returning a 404 to the user.
I don't think this is a bug, but I believe that 404 response is simply
better and more obvious than 500 in this case and since Blobstore API is
still "experimental" and "you were eager to get my feedback so that you
could improve the API", I think it's good time to talk about this now ;)
Just take a look at competing solutions that work this way:
- S3 returns 404 for NoSuchBucket and NoSuchKey errors [1],
- Azure Blob Service returns 404 for ContainerNotFound and BlobNotFound
errors [2],
- nginx returns 404 when file pointed by "X-Accel-Redirect" doesn't exist.
[1]
http://docs.amazonwebservices.com/AmazonS3/latest/API/index.html?ErrorCodeList.html
[2] http://msdn.microsoft.com/en-us/library/dd179439.aspx