Advice on File Upload in otherwise JSONRPC API

[michael kariv]

Hi all, need your advice. I have a nice JSONRPC (over http) based web
service in the making. However I need just one call it to be able to
upload a file. What you, as API interface designers would do:
- and one exception, simple POST based upload?
- make client read the content of the file and send the blob as a
parameter to a JSON-RPC API call?
- something else?

I am not sure I am the first to stumble upon the use case. Gurus out
there, please do help.

Michael

[Skylos]

Hmm, interesting application.

Since files should be streamed, as they could be enormous and not fitting in memory, I'd want to keep it out of the json stream because the json object itself is probably going to be loaded completely in memory.  Maybe something like...

==> POST JSON "I want to upload a file called X"

<== STATUS 200 OK "PUT it at URL/X"
==> PUT URL/X
<== STATUS 200 OK

And program your PUT verb to hit the upload handler.  That stays semantically reasonable in the realm of HTTP protocols and such, and there's no reason your little client couldn't speak PUT as well as POST, right?

Of course, if they were just little text files or something, I'd say shove them into the json.  binary and large files, though, I'd rather see transfered 'out of band ' so the json-rpc system doesn't have to deal with the logistics of large

My take,

David

[Stephen McKamey]

I'm not sure what your client is, but a web browser even with the most
ninja of JavaScript skills can only post files via "multipart/form-
data" encoding due to tight security around files. I've written a
parser for this encoding and it was about as incompatible with JSON-
RPC as you could imagine (in case you're thinking of jamming it in
there).

Of course if your client isn't a browser (including a Flash plug-in)
then the only major issue is what to choose as your binary encoding.
I'd agree with David though that JSON-RPC isn't going to mesh well
with the best practices around large file handling (e.g. streaming).

smm

[Gaetano Giunta]

Stephen McKamey a écrit :

> I'm not sure what your client is, but a web browser even with the most
> ninja of JavaScript skills can only post files via "multipart/form-
> data" encoding due to tight security around files. I've written a
> parser for this encoding and it was about as incompatible with JSON-
> RPC as you could imagine (in case you're thinking of jamming it in
> there).
>
> Of course if your client isn't a browser (including a Flash plug-in)
> then the only major issue is what to choose as your binary encoding.
> I'd agree with David though that JSON-RPC isn't going to mesh well
> with the best practices around large file handling (e.g. streaming).
>
>

Size pretty much dictates the approach here, as the memory and time
spent to base64_encode your file and the json_encode it might be too
much of an overhead, both in cpu time and memory usage (depending of
course on the json toolkit used).
If big file sizes are not accepted by the system anyway, I'd say it
would be worth saving the hassle of providing a 2nd protocol for uploads.

Gaetano

[Roland Koebler]

hi,

> Since files should be streamed, as they could be enormous and not fitting in
> memory, I'd want to keep it out of the json stream

yes. small binary data may fit into json and json-rpc; but for large blobs
of data it's probably sensible to send them as a "raw" stream.

so, I would suggest to tell the server that you want to upload some data
by calling a json-rpc-method, which replies with an
url/ip+port/etc. where to upload the data, e.g.

--> {"jsonrpc": "2.0", "method": "send_image", "params": ..., "id": 1}
<-- {"jsonrpc": "2.0", "result": {"URL": "/exampleurl", "maxsize": 10000000, "accepted-format":["jpg", "png"]}, "id": 1}

--> send data directly to the remote server

and maybe to check if the data arrived correctly:
--> {"jsonrpc": "2.0", "method": "send_done", "params": {"checksum": "1a5e8f13"}, "id": 2}
<-- {"jsonrpc": "2.0", "result": "ok"}

I think that's the most simple and cleanest solution.

regards,
Roland

[Joseph Scott]

I'd agree with some sort of technique that allows you to simply post
the data to a URL. We occasionally get reports of memory problems
when people upload large files in WordPress via XML-RPC, the base64
encoding/decoding can be a beast.

--
Joseph Scott
jos...@josephscott.org
http://josephscott.org/

[michael kariv]

All,
Thank you guys. I think we have a concensus.
All kinds of applications of my Web Service I can't predict, but the
client in mind is not a browser.
Web protocols are not only for browsers nowadays.

The approach of asking for an upload handle is perfect. The more I
think about it the more I like it. It blends well with my overall
approach.

Thanks to the group again.

Michael

> jos...@josephscott.orghttp://josephscott.org/- Hide quoted text -
>
> - Show quoted text -