Large data in requests/replies

[marko.b...@gmail.com]

What is the right way to send large data (for example JPEG files with the size up to 5 MBytes) from client to server and back inside the JSON over HTTP connection?

I've tried to encode the files to base64 and send them as stings (one file in the request/reply), but it is not working whether I send data to or from server.

Everything else is working fine.

[Matt Broadstone]

Hi Marko,

Yeah encoding it as base64 string would be the way to go here, although its incredibly inefficient. If you can submit a PR with a test for data this large I can take a look at why its not working (though I suspect there is a limit in QJsonDocument maybe). Also, please specify which version of Qt and qjsonrpc you are using.

Thanks!

M

[marko.b...@gmail.com]

Thank you for the reply!

> Yeah encoding it as base64 string would be the way to go here,
> although its incredibly inefficient.

What is the right way to send binary data with JSON RPC?

May be it is possible to process file transfers over GET/POST with the QJsonRpcHttpServer?

> If you can submit a PR with a test for data this large
> I can take a look at why its not working

Thank you, but the project is to large to pull out the example.

> though I suspect there is a limit in QJsonDocument maybe

May be. The short strings (up to 15K symbols) are transmitting fine. If there is no way to send binary data with JSON RPC except base64, I'll implement the HTTP server for sending/receiving files which will be working on alternative port.

Thank you,
Marko.

[Matt Broadstone]

On Sat, Mar 28, 2015 at 1:57 PM, <marko.b...@gmail.com> wrote:

Thank you for the reply!

> Yeah encoding it as base64 string would be the way to go here,
> although its incredibly inefficient.

What is the right way to send binary data with JSON RPC?

Unfortunately the only way to send binary data with jsonrpc is to use base64 encoded strings. It's very inefficient, I would recommend researching alternative ways to transfer your files (perhaps an RPC command to trigger a "fetch" of a url using QNetworkAccessManager on the "server" side?)

May be it is possible to process file transfers over GET/POST with the QJsonRpcHttpServer?

QJsonRpcHttpServer is specialized for jsonrpc calls, it wouldn't be easy (or in line with the projects goals) to modify it to also serve arbitrary files. You are better off setting up your own e.g. http server to serve the files you want to transfer.

 

> If you can submit a PR with a test for data this large
> I can take a look at why its not working

Thank you, but the project is to large to pull out the example.

Having said all of that, it _should_ be possible to send arbitrarily large base64 strings over jsonrpc using qjsonrpc. So this sounds like a bug you've run into. I'm not saying you need to commit the whole project, but if you could make a small test case that emulates this behavior inside the qjsonrpc auto tests (add a 5mb jpeg file, read it in with QFile, base64 encode and send the QJsonRpcmessage) that would be really appreciated. I'm sure there are other people that have run into this issue.

 

> though I suspect there is a limit in QJsonDocument maybe

May be. The short strings (up to 15K symbols) are transmitting fine. If there is no way to send binary data with JSON RPC except base64, I'll implement the HTTP server for sending/receiving files which will be working on alternative port.

Yes I think this is the bug I was mentioning. Again, since we are working with JSON here there are no defined binary types, so base64 encoded strings have to be the way to go.

 

Thank you,
Marko.

Cheers,

M

 

--
You received this message because you are subscribed to the Google Groups "qjsonrpc-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to qjsonrpc-develop...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[marko.b...@gmail.com]

> Having said all of that, it _should_ be possible to send arbitrarily large
> base64 strings over jsonrpc using qjsonrpc. So this sounds like a bug
> you've run into. I'm not saying you need to commit the whole project, but
> if you could make a small test case that emulates this behavior inside
> the qjsonrpc auto tests (add a 5mb jpeg file, read it in with QFile,
> base64 encode and send the QJsonRpcmessage) that would be really
> appreciated. I'm sure there are other people that have run into this issue.

Ok. I'll do it next week.

Thank you for your excellent work.

Marko.