Large data amounts via. RESTful API

[Claus Munch]

Hi group!

I was wondering if any of you have some input on uploading large data sets (eg. large log files, trace files, stack dumps etc.) via. a RESTful API?

Any input on how this is best achieved?

The scenario is, I want to post data from both Windows and *nix systems to the same API, but I am (primarely) limited to using HTTP requests against the API.

Any guidelines, links to articles or other?

Thanks!

/Claus

[Ashwin Purohit]

If you don't want to post huge files at once, you could split up the data file into small chunks. For each chunk send a separate HTTP POST request, specifying the bytes of the original file it represents, in the Range header. When the server receives all chunks (it knows the total size of the uploaded file because you sent a delimiter), it knows the file upload is complete and can concatenate it server side, perhaps returning a Content-MD5 (or ETag, as is often done) so you can verify the file.

This of course requires server-side cooperation, because it's non-standard.

[Matt Jones]

We are using HTTP Transfer-Encoding: Chunked to achieve this objective.  It's one POST, but it gets streamed to the code for each chunk, so it doesn't end up consuming a ton of memory on the server.