So it works fine with files up to 50K (around that size, not exact), where smaller files always upload and larger files always fail. I do use URL-safe base64 encoding so that cannot be the problem. Also, I've tried several mime-types and that seems to be of no influence.
At last I found out why upload fails. Apparently this method posts data inline and that is limited to 64K. So I should use the *media upload feature* of the JSON API. I cannot work out how to make the javascript client do that (I've posted a question on https://groups.google.com/forum/?fromgroups#!forum/google-api-javascr... that end).
> So it works fine with files up to 50K (around that size, not exact), where > smaller files always upload and larger files always fail. I do use URL-safe > base64 encoding so that cannot be the problem. Also, I've tried several > mime-types and that seems to be of no influence.
The use of the Farm API seems to me to add needless confusion to the whole thing, but the upload uri is actually documented on that page:
> *The upload URI.* The format of the upload endpoint is the standard > resource URI with an “upload” prefix. Use this URI when transferring the > media data itself. Example: POST /upload/farm/v1/animals
On Thursday, September 27, 2012 5:08:19 AM UTC+10, joop.ringelberg wrote:
> At last I found out why upload fails. Apparently this method posts data > inline and that is limited to 64K. > So I should use the *media upload feature* of the JSON API. > I cannot work out how to make the javascript client do that (I've posted a > question on > https://groups.google.com/forum/?fromgroups#!forum/google-api-javascr... that end).
>> So it works fine with files up to 50K (around that size, not exact), >> where smaller files always upload and larger files always fail. I do use >> URL-safe base64 encoding so that cannot be the problem. Also, I've tried >> several mime-types and that seems to be of no influence.
I've written a Javascript program that successfully uses the media upload
feature, and I've verified it with files up to 10MB. I also bumped into the
doc problem Glenn noted about the need to prefix the URL with '/upload' and
wrote an internal request to fix that (thanks Glenn!).
I found this article on the JavaScript file
api<http://www.html5rocks.com/en/tutorials/file/dndfiles/>very
helpful. My current version of this code loads the file contents into
memory, which doesn't work well for large files but in the near future I'll
clean this up, add file slicing/multipart upload and share a more complete
sample app. In the meantime, hopefully the snippet below will help unblock
you but let me know if you have any questions.
Attached is the most important part of my code - my file API reader's
onload event handler, which does the upload to Cloud Storage.
> The use of the Farm API seems to me to add needless confusion to the whole
> thing, but the upload uri is actually documented on that page:
>> *The upload URI.* The format of the upload endpoint is the standard
>> resource URI with an “upload” prefix. Use this URI when transferring the
>> media data itself. Example: POST /upload/farm/v1/animals
>>> So it works fine with files up to 50K (around that size, not exact),
>>> where smaller files always upload and larger files always fail. I do use
>>> URL-safe base64 encoding so that cannot be the problem. Also, I've tried
>>> several mime-types and that seems to be of no influence.
Thanks for the extensive reply. I've tried it and can store objects using your code (well, embedded in my own code, see below). However, when I retrieve the object from storage, there is no "data" property in the media.
This is the response when I build the request using gapi.client.storage.
objects.insert:
[
{
"id": "gapiRpc",
"result": {
"kind": "storage#object",
"id": "ttf-7-json/test1.json",
"name": "test1.json",
"bucket": "ttf-7-json",
"media": {
"contentType": "application/json",
"timeCreated": "2012-09-27T20:34:48.858Z",
"length": "9",
"hash": "48f604dbf6942f0e02986542641ed635",
"algorithm": "MD5",
"data": "e2FhcCA6IDF9"
On retrieving the object, the data property is missing, too.
This probably is in accordance with the documentation of the object resource:
URL-safe Base64-encoded <https://tools.ietf.org/html/rfc4648#section-5> data. This property can be used to insert objects under 64KB in size, and will only be returned in response to the get method for objects so created. When this resource is returned in response to the list method, this property is omitted.
Now this may be a dumb question, but just how do I retrieve the data if uploaded using media upload? It is there, I can access it through the webinterface at https://storage.cloud.google.com, but requests built with gapi.client.storage.objects.get do not return it.
Thanks,
Joop
On Thursday, September 27, 2012 8:28:17 AM UTC+2, Google Cloud Storage Team wrote:
> I've written a Javascript program that successfully uses the media upload > feature, and I've verified it with files up to 10MB. I also bumped into the > doc problem Glenn noted about the need to prefix the URL with '/upload' and > wrote an internal request to fix that (thanks Glenn!).
> I found this article on the JavaScript file api<http://www.html5rocks.com/en/tutorials/file/dndfiles/>very helpful. My current version of this code loads the file contents into > memory, which doesn't work well for large files but in the near future I'll > clean this up, add file slicing/multipart upload and share a more complete > sample app. In the meantime, hopefully the snippet below will help unblock > you but let me know if you have any questions.
> Attached is the most important part of my code - my file API reader's > onload event handler, which does the upload to Cloud Storage.
>> The use of the Farm API seems to me to add needless confusion to the >> whole thing, but the upload uri is actually documented on that page:
>>> *The upload URI.* The format of the upload endpoint is the standard >>> resource URI with an “upload” prefix. Use this URI when transferring the >>> media data itself. Example: POST /upload/farm/v1/animals
>>>> So it works fine with files up to 50K (around that size, not exact), >>>> where smaller files always upload and larger files always fail. I do use >>>> URL-safe base64 encoding so that cannot be the problem. Also, I've tried >>>> several mime-types and that seems to be of no influence.
Ah well, I should not be doing these things late in the evening!
It was a dumb question, after all.
After configuring CORS on the bucket I could retrieve the resource using straightforward AJAX. Of course.
One more thing, though: I noticed that you use btoa() to base64 encode the data. I used to apply an URL safe version, which, to my knowledge, the btoa() function does not supply. However, that URL safe version led to a complaint of the server that the multipart body was malformed. I then used the btoa() function like in your code and it worked.
To sum it up, is it correct that:
* using object.insert, the media.data content should be URL safe encoded
* doing media upload, plain base64 is required?
On Thursday, September 27, 2012 10:51:36 PM UTC+2, joop.ringelberg wrote:
> Hi Marc,
> Thanks for the extensive reply. I've tried it and can store objects using > your code (well, embedded in my own code, see below). However, when I > retrieve the object from storage, there is no "data" property in the media.
> And here is the response I get when using your code (when the request is > built using gapi.client.request):
> {
> "kind": "storage#object",
> "id": "ttf-7-json/test2.json",
> "selfLink": "
> https://www.googleapis.com/storage/v1beta1/b/ttf-7-json/o/test2.json?... > ",
> "name": "test2.json",
> "bucket": "ttf-7-json",
> "media": {
> "contentType": "application/json",
> "timeCreated": "2012-09-27T20:34:49.444Z",
> "length": "9",
> "hash": "48f604dbf6942f0e02986542641ed635",
> "algorithm": "MD5",
> "link": "
> https://www.googleapis.com/storage/v1beta1/b/ttf-7-json/o/test2.json"
> },
> "owner": {
> "entity": > "user-00b4903a972c73be135e58c61104242e59426dad3f655daf103301d1e99f065a",
> "entityId": > "00b4903a972c73be135e58c61104242e59426dad3f655daf103301d1e99f065a"
> }
> }
> Notice the missing "data" property.
> On retrieving the object, the data property is missing, too.
> This probably is in accordance with the documentation of the object > resource:
> URL-safe Base64-encoded <https://tools.ietf.org/html/rfc4648#section-5> data. > This property can be used to insert objects under 64KB in size, and will > only be returned in response to the get method for objects so created. When > this resource is returned in response to the list method, this property is > omitted.
> Now this may be a dumb question, but just how do I retrieve the data if > uploaded using media upload? It is there, I can access it through the > webinterface at https://storage.cloud.google.com, but requests built with > gapi.client.storage.objects.get do not return it.
> Thanks,
> Joop
> On Thursday, September 27, 2012 8:28:17 AM UTC+2, Google Cloud Storage > Team wrote:
>> Hi Joop,
>> I've written a Javascript program that successfully uses the media upload >> feature, and I've verified it with files up to 10MB. I also bumped into the >> doc problem Glenn noted about the need to prefix the URL with '/upload' and >> wrote an internal request to fix that (thanks Glenn!).
>> I found this article on the JavaScript file api<http://www.html5rocks.com/en/tutorials/file/dndfiles/>very helpful. My current version of this code loads the file contents into >> memory, which doesn't work well for large files but in the near future I'll >> clean this up, add file slicing/multipart upload and share a more complete >> sample app. In the meantime, hopefully the snippet below will help unblock >> you but let me know if you have any questions.
>> Attached is the most important part of my code - my file API reader's >> onload event handler, which does the upload to Cloud Storage.
>>> The use of the Farm API seems to me to add needless confusion to the >>> whole thing, but the upload uri is actually documented on that page:
>>>> *The upload URI.* The format of the upload endpoint is the standard >>>> resource URI with an “upload” prefix. Use this URI when transferring the >>>> media data itself. Example: POST /upload/farm/v1/animals
>>> On Thursday, September 27, 2012 5:08:19 AM UTC+10, joop.ringelberg wrote:
>>>> At last I found out why upload fails. Apparently this method posts data >>>> inline and that is limited to 64K. >>>> So I should use the *media upload feature* of the JSON API. >>>> I cannot work out how to make the javascript client do that (I've >>>> posted a question on https://groups.google.com/** >>>> forum/?fromgroups#!forum/**google-api-javascript-client<https://groups.google.com/forum/?fromgroups#!forum/google-api-javascr...>to that end).
