JSON-RPC *not* over HTTP?
sambayer
I downloaded and began to play with the Python json-rpc implementation
developed by Ronald Koebler, announced on this list on 9/2, and since
he was the original author of the 2.0 specification, I assumed that it
was a reference implementation. But it took me a bit of time to digest
the fact that the protocol is independent of the transport, and that
the client/server implementation in Ronald's module doesn't have any
official status.
Which is good, because it has a pretty severe bug, namely that the
length of the message is explicitly limited, on the server side. The
way TCP/IP works, as I understand it, it's just not possible to write
a robust client/server implementation the way he's set it up; you
either need to send the length of the message in advance of the
message, or terminate the message in some distinguished way (e.g., a
newline). Otherwise, the server can't know when it should believe it's
done reading a message.
This leads me to wonder: has anyone tried to define a "standard" "raw"
transport for JSON-RPC? The overhead of running a Web server just to
set up JSON-RPC services is too great for my purposes, and I'm
wondering whether I can expect any of the programming language
packages to agree on a standard stand-alone protocol.
Thanks in advance -
Sam Bayer
The MITRE Corporation
s...@mitre.org
Roland Koebler
> and since
> he was the original author of the 2.0 specification, I assumed that it
> was a reference implementation.
no, it isn't a reference implementation. it's still beta, and it only
implements what I currently need. everything I don't use is pretty much
untested. (and I mainly use jsonrpc over unix domain sockets.)
and since JSON-RPC 2.0 isn't released yet, there currently is no
reference implementation.
> because it has a pretty severe bug, namely that the
> length of the message is explicitly limited, on the server side.
hmm, where is the limit you found?
or do you mean the "limit"-parameter?
> The
> way TCP/IP works, as I understand it, it's just not possible to write
> a robust client/server implementation the way he's set it up; you
> either need to send the length of the message in advance of the
> message, or terminate the message in some distinguished way (e.g., a
> newline). Otherwise, the server can't know when it should believe it's
> done reading a message.
that's not quite right.
- my implemented TCP-transport (which is based on the socket-module)
has a "limit"-parameter, which tells the socket how much bytes
to receive. of course, all messages which are greater than this
socket-limit result in an error.
- the "socket-limit" maybe is restricted to some max. value (maybe 64K).
so, my implementation currently is not suited for applications where
larger messages are needed.
- but with an appropriate "limit", my implementation worked really well
and robustly in all of my tests. please tell me if it fails in your
tests.
- my implementation also (currently) only uses 1 message per connection,
and closes the connection afterwards.
- separating multiple requests on the same connection is quite an
other "problem". this can be solved in several ways:
- using a streaming json-parser, which detects when a json-string
is done. If you don't have a streaming json-parser, you can use
a very simple json-"splitter". I already wrote such a json-splitter
in python, but it's not (yet?) integrated into my json-rpc-
implementation.
- using a transport which supports the message-separation.
This could be HTTP; but some "trivial" protocol which first
sends the message-length and then the message should work
as well.
- don't use such things as "newlines" to separate messages. never.
(since newlines can also appear in a message.)
> This leads me to wonder: has anyone tried to define a "standard" "raw"
> transport for JSON-RPC?
no, currently noone here has defined a transport like "message_length +
message over TCP".
most people here probably use http; and I'm using raw sockets
(unix domain sockets or TCP), with 1 message per connection and
relatively small messages, which works really well for me.
If you need something else (multiple sockets over 1 TCP-connection?
unlimited length? etc.), please tell me what exactly you think
you need, and maybe I'll implement it.
> The overhead of running a Web server just to
> set up JSON-RPC services is too great for my purposes,
definitely.
regards,
Roland
sambayer
> > because it has a pretty severe bug, namely that the
> > length of the message is explicitly limited, on the server side.
>
> hmm, where is the limit you found?
> or do you mean the "limit"-parameter?
> > This leads me to wonder: has anyone tried to define a "standard" "raw"
> > transport for JSON-RPC?
>
> no, currently noone here has defined a transport like "message_length +
> message over TCP".
>
> most people here probably use http; and I'm using raw sockets
> (unix domain sockets or TCP), with 1 message per connection and
> relatively small messages, which works really well for me.
> If you need something else (multiple sockets over 1 TCP-connection?
> unlimited length? etc.), please tell me what exactly you think
> you need, and maybe I'll implement it.
listed: adding a newline delimiter. That's easily implemented in your
module. I'm interoperating with a co-worker who's using his own OCaml
implementation, and that's what he picked. Personally, I'd prefer
message length + message, but we don't seem to need that locally yet.
My curiosity about a "raw" transport was in the interest of
standardization - I couldn't tell whether I was missing something.
JSON-RPC is such a clean, simple protocol, it would be nice if the
various emerging implementations had something lightweight to converge
on in the absence of living inside HTTP.
Thanks again.
Sam Bayer
Roland Koebler
> > or do you mean the "limit"-parameter?
> Yes, that's the limit I was referring to.
since my messages always were "small" (< 64kb) this limit wasn't a
problem for me. but I'll probably add my json-splitter to my
json-rpc-implementation soon, so that it should also work with
arbitrary lengths.
> Our local adaptation, so far, has been one of the alternatives you
> listed: adding a newline delimiter. That's easily implemented in your
> module. I'm interoperating with a co-worker who's using his own OCaml
> implementation, and that's what he picked.
but note that this can only work if you make *absolutely sure* that there
are no newlines in the json-data!!
> Personally, I'd prefer
> message length + message, but we don't seem to need that locally yet.
I'd prefer my json-splitter-approach (=using a very simple kind of
json-parser, which only recognizes when a json-object is done), since
this always works, without restricting the data (like "no newlines")
and without introducing a new protocol.
> My curiosity about a "raw" transport was in the interest of
> standardization - I couldn't tell whether I was missing something.
> JSON-RPC is such a clean, simple protocol, it would be nice if the
> various emerging implementations had something lightweight to converge
> on in the absence of living inside HTTP.
ok, thanks for your note. I think I'll write something about this
on my json-rpc-page (and probably propose to use a json-splitter,
incl. code-examples).
regards,
Roland
sambayer
> > Personally, I'd prefer
> > message length + message, but we don't seem to need that locally yet.
>
> I'd prefer my json-splitter-approach (=using a very simple kind of
> json-parser, which only recognizes when a json-object is done), since
> this always works, without restricting the data (like "no newlines")
> and without introducing a new protocol.
other programming languages, since my interest is not simply
interoperating with other Python services, but also Java, OCaml and
possibly Common Lisp.
Cheers,
Sam Bayer
Roland Koebler
> > I'd prefer my json-splitter-approach (=using a very simple kind of
> > json-parser, which only recognizes when a json-object is done), since
> > this always works, without restricting the data (like "no newlines")
> > and without introducing a new protocol.
>
> My concern about this is how hard or easy it would be to implement in
> other programming languages
it should be very easy to implement in every programming language.
the only thing such a "splitter" has to do is to walk through the
string and look for {[]}\".
this was discussed some time ago on this list, and you could look
in the archive if you're interested. I also posted a quickhack-example
of a json-splitter there.
(->see "Re: Why is 2.0 client-server and not peer-to-peer?",
esp. my mails on 2008-09-20)
regards,
Roland