Uploading files to OneDrive with chunked Transfer-Encoding

[jensant...@gmail.com]

Hi Shai,

My app is trying to upload files to OneDrive through REST API. It works fine with small files, but it throws some memory or Heap Space errors when uploading large files in Android. Soon I have found that it is the problem of RetryableOutputStream, and the fix is to add header "Transfer-Encoding: chunked" to the request.

It stops the above error but deduce another set of problems:
For Android, the large files can be uploaded successfully, but when it finishes uploads, it prompts "Read timed out" Exception. If I press "retry", it will overwrite the uploaded file with 0 bytes; otherwise if I press "cancel", everything looks alright.
For small files, there is no problem at all.

For iOS, it gives "request timed out" for large files but the file have not been uploaded.
For small files, it gives "400 Bad Request" error.

The differences in behaviors make me confused and I don't know what the problem is. Could you shed some lights on this issue? Thank you.

[Shai Almog]

Hi,
are you using the multipart upload or just writing the file using post?
Upload is a pretty tricky thing, I suggest going with the multipart class.

[jensant...@gmail.com]

Yes, I have tried both PUT (which is very easy) and POST (with multipart). Both methods are working perfectly, without the Transfer-Encoding header...

Whenever I add the Transfer-Encoding header, the same problem occurs.



Shai Almog於 2014年12月17日星期三UTC+8下午2時44分27秒寫道：

[jensant...@gmail.com]

Is it possible that there is somewhere ConnectionRequest has added the "Content-Length" header?
Adding the "Content-Length" will fail the "Transfer-Encoding: chunked" header.

[Shai Almog]

The multipart request does add the content length. You can look at its source code in our open source project: http://code.google.com/p/codenameone

[jensant...@gmail.com]

Just found this in the MultipartRequest (https://code.google.com/p/codenameone/source/browse/trunk/CodenameOne/src/com/codename1/io/MultipartRequest.java):

protected void initConnection(Object connection) {
        contentLength = calculateContentLength();
        addRequestHeader("Content-Length", Long.toString(contentLength));
        super.initConnection(connection);
}

There is a line which add the content length when initConnection.


But anyway, I have tried to write my own Multipart request which doesn't have Content-Length, and it gives the results mentioned above. Without the "transfer-encoding: chunked", it works.

[Shai Almog]

Multipart does need the content length as I stated above.
Check out this article on debugging networking calls using low level tools: http://www.codenameone.com/debugging-webservice-calls.html

[jensant...@gmail.com]

Thanks for the advice.

I have used a proxy to debug the problems, and have found that for Android, there is a "Content-length: " header added, where for iOS there is "Content-length: xxx" added. They are using the same code however.
So it seems that iOS will eventually add the Content-length thing somewhere, which fails the "Transfer-Encoding: chunked"......

For the timeout thing in Android, it is just the response was too slow. It should be able to be fixed by setting timeout or add some wait methods.

So is there anyway to remove the content-length? Is it a native behavior? Any suggestions? Thanks a lot.

[Shai Almog]

If it behaves differently between Android/iOS/SE this seems like something from the native in iOS which does all sorts of odd things (like gzipping content etc.).
I'm not sure if there is a workaround for something like that and it would require some complex investigation.
I suggest you try to verify that this is indeed the actual problem by creating that header manually and seeing is it reproduces the exact problem in the simulator.