upload failure for file size above 50K

[joop.ringelberg]

Hi,

using the google-api-javascript-client, I find that upload of files larger than around 50K fails. Here is the code:

                var request = gapi.client.storage.objects.insert(
                                {
                                    'bucket': bucket,
                                    'name': fileName,
                                    'resource': {
                                                "media": {
                                                            "contentType": mimeType,
                                                            "contentLength": data.length,
                                                            "data": data
                                                        },
                                                'cacheControl': 'no-cache'
                                            }
                                    });

And here is the response:

[

{

"error": {

"code": 400,

"message": "Invalid argument.",

"data": [

{

"domain": "global",

"reason": "invalid",

"message": "Invalid argument."

}

]

},

"id": "gapiRpc"

}

]

The payload:

[{"jsonrpc":"2.0","id":"gapiRpc","method":"storage.objects.insert","params":{"bucket":"ttf-7-json","name":"d7710050-a701-408b-a07e-001cec0a9494.jpeg","resource":{"media":{"contentType":"image/jpeg","contentLength":107132,"data":"_9j_4AAQSkZJRgABAQAAAQABAAD__gA8Q1JFQVRPUjogZ2QtanBlZyB2MS4wICh1c2luZyBJSkcgSlBFRyB2NjIpLCBxdWFsaXR5ID0gMTAwCv_bAEMAAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQE

...

59vzx_Q0UYz6_mR_I0UD17_AIf15_0tf__Z"},"cacheControl":"no-cache"}},"apiVersion":"v1beta1"}]

(data field abbreviated).

So it works fine with files up to 50K (around that size, not exact), where smaller files always upload and larger files always fail. I do use URL-safe base64 encoding so that cannot be the problem. Also, I've tried several mime-types and that seems to be of no influence.

The upload guide (https://developers.google.com/storage/docs/json_api/v1/how-tos/upload#resumable) states that for files above 5Mb one should switch to resumable upload. However, 50K is quite a bit less and the error seems not to indicate a problem with interrupted transfer.

Any ideas?

Thanks in advance,

Joop

[joop.ringelberg]

At last I found out why upload fails. Apparently this method posts data inline and that is limited to 64K. 

So I should use the media upload feature of the JSON API. 

I cannot work out how to make the javascript client do that (I've posted a question on https://groups.google.com/forum/?fromgroups#!forum/google-api-javascript-client to that end).

I also tried to do an ordinary POST request using AJAX. However, I cannot work out the correct endpoint. The documentation on https://developers.google.com/storage/docs/json_api/v1/how-tos/upload mentions a fictional Farm API and says:

However the same concepts apply to the Google Cloud Storage JSON API.

But how?

Joop

[Glenn Fowler]

One critical thing you might be missing (it certainly held me up for a while) is that the base url changes - instead of posting to https://www.googleapis.com/storage/v1beta1/b/bucketname/o?uploadType=type you need to post to https://www.googleapis.com/upload/storage/v1beta1/b/bucketname/o?uploadType=type - that 'upload' at the start is very important...

The use of the Farm API seems to me to add needless confusion to the whole thing, but the upload uri is actually documented on that page:

The upload URI. The format of the upload endpoint is the standard resource URI with an “upload” prefix. Use this URI when transferring the media data itself. Example: POST /upload/farm/v1/animals

Also if you're handling resumable uploads the documentation still hasn't been updated (per https://groups.google.com/forum/?fromgroups=#!topic/gs-discussion/fNr1C_Fl0gM ) so that missing 'bytes' is something to watch for.

[Google Cloud Storage Team]

Hi Joop,

I've written a Javascript program that successfully uses the media upload feature, and I've verified it with files up to 10MB. I also bumped into the doc problem Glenn noted about the need to prefix the URL with '/upload' and wrote an internal request to fix that (thanks Glenn!).

I found this article on the JavaScript file api very helpful. My current version of this code loads the file contents into memory, which doesn't work well for large files but in the near future I'll clean this up, add file slicing/multipart upload and share a more complete sample app. In the meantime, hopefully the snippet below will help unblock you but let me know if you have any questions.

 

Attached is the most important part of my code - my file API reader's onload event handler, which does the upload to Cloud Storage.

Marc

Google Cloud Storage Team

  reader.onload = function(e) {   
    var fileData = evt.target.files[0];
    const boundary = '-------314159265358979323846';
    const delimiter = "\r\n--" + boundary + "\r\n";
    const close_delim = "\r\n--" + boundary + "--";
    var contentType = fileData.type || 'application/octet-stream';
    var metadata = {
      'title': fileData.name,
      'mimeType': contentType
    };
    var base64Data = btoa(reader.result);
    var multipartRequestBody =
        delimiter +
        'Content-Type: application/json\r\n\r\n' +
        JSON.stringify(metadata) +
        delimiter +
        'Content-Type: ' + contentType + '\r\n' +
        'Content-Transfer-Encoding: base64\r\n' +
        '\r\n' +
        base64Data +
        close_delim;
    var request = gapi.client.request({
        'path': '/upload/storage/v1beta1/b/marc-us/o',
        'method': 'POST',
        'params': {
                    'name': fileData.name,
                    'uploadType': 'multipart'
                  },
        'headers': {
          'Content-Type': 'multipart/mixed; boundary="' + boundary + '"'
        },
        'body': multipartRequestBody});
    callback = function() { alert('file uploaded'); };
    request.execute(callback);
  }
  reader.readAsBinaryString(evt.target.files[0]);
}

--
 
 

--
Marc Cohen

Google Cloud Storage Team

[joop.ringelberg]

Hi Marc,

Thanks for the extensive reply. I've tried it and can store objects using your code (well, embedded in my own code, see below). However, when I retrieve the object from storage, there is no "data" property in the media.

This is the response when I build the request using gapi.client.storage.objects.insert:

[

{

"id": "gapiRpc",

"result": {

"kind": "storage#object",

"id": "ttf-7-json/test1.json",

"name": "test1.json",

"bucket": "ttf-7-json",

"media": {

"contentType": "application/json",

"timeCreated": "2012-09-27T20:34:48.858Z",

"length": "9",

"hash": "48f604dbf6942f0e02986542641ed635",

"algorithm": "MD5",

"data": "e2FhcCA6IDF9"

},

"cacheControl": "no-cache",

"owner": {

"entity": "user-00b4903a972c73be135e58c61104242e59426dad3f655daf103301d1e99f065a",

"entityId": "00b4903a972c73be135e58c61104242e59426dad3f655daf103301d1e99f065a"

}

}

}

]

And here is the response I get when using your code (when the request is built using gapi.client.request):

{

"kind": "storage#object",

"id": "ttf-7-json/test2.json",

"selfLink": "https://www.googleapis.com/storage/v1beta1/b/ttf-7-json/o/test2.json?alt=json",

"name": "test2.json",

"bucket": "ttf-7-json",

"media": {

"contentType": "application/json",

"timeCreated": "2012-09-27T20:34:49.444Z",

"length": "9",

"hash": "48f604dbf6942f0e02986542641ed635",

"algorithm": "MD5",

"link": "https://www.googleapis.com/storage/v1beta1/b/ttf-7-json/o/test2.json"

},

"owner": {

"entity": "user-00b4903a972c73be135e58c61104242e59426dad3f655daf103301d1e99f065a",

"entityId": "00b4903a972c73be135e58c61104242e59426dad3f655daf103301d1e99f065a"

}

}

Notice the missing "data" property.

On retrieving the object, the data property is missing, too.

This probably is in accordance with the documentation of the object resource:

URL-safe Base64-encoded data. This property can be used to insert objects under 64KB in size, and will only be returned in response to the get method for objects so created. When this resource is returned in response to the list method, this property is omitted.

Now this may be a dumb question, but just how do I retrieve the data if uploaded using media upload? It is there, I can access it through the webinterface at https://storage.cloud.google.com, but requests built with gapi.client.storage.objects.get do not return it.

Thanks,

Joop

[joop.ringelberg]

Ah well, I should not be doing these things late in the evening!

It was a dumb question, after all.

After configuring CORS on the bucket I could retrieve the resource using straightforward AJAX. Of course.

One more thing, though: I noticed that you use btoa() to base64 encode the data. I used to apply an URL safe version, which, to my knowledge, the btoa() function does not supply. However, that URL safe version led to a complaint of the server that the multipart body was malformed. I then used the btoa() function like in your code and it worked.

To sum it up, is it correct that:

* using object.insert, the media.data content should be URL safe encoded

* doing media upload, plain base64 is required?

Thanks again.

Joop