JSON-RPC error recovery

[David Lee]

Hello, all!

We're considering using JSON-RPC for a new project, and I had some questions about the protocol. We have a lot of asynchronous events that go between the server and client, so we're primarily interested in JSON-RPC over TCP sockets. In reading the spec, I had some questions about error recovery.

For example, from the spec:

--> {"jsonrpc": "2.0", "method": "foobar, "params": "bar", "baz]
<-- {"jsonrpc": "2.0", "error": {"code": -32700, "message": "Parse error."}, "id": null}

I can see where the server can detect the parse error and respond appropriately. But what then? Since TCP is a streaming protocol, how would the server resync back with the client and start processing requests again?

Similarly, how would it handle a truncated message?

--> {"jsonrpc": "2.0", "method": "foo", "params": [1, 2,

How would the server know that the client intended for this to be a complete message? If the client were to continue sending requests, wouldn't the server simply continue to build the params for invoking the method foo?

Thanks!

-- 

dave

<><

[Matt (MPCM)]

Hi,

The server's job is to accept strings, break them into chunks via something like (End Of Transmission) 0x04, parse those into valid json objects and then into valid json-rpc requests to be handled. So... unless something like EOT is sent (there are old posts discussing separator/EOT approaches), the question then becomes how large of a buffer will server use before it throws an error filled response object.

If the server is doing partial object building as the string goes along, it could know early on that the parse will fail.... Even then, it needs to be able to junk any additional input until it can make a clean start. At some point the server will need to give up, without a clean truncation point.

If you are dealing one stream per client/server connection, would it be valid approach to terminate the connection?

Hopefully others using/writing streaming clients more regularly than me can comment on their approaches. Generally this is only happening when the json is not being generated by an encoder directly, perhaps a client spewing bad data?

--
Matt (MPCM)

[Lorenzo Pastrana]

EOT ... I'm by no mean a stream / bare tcp expert but this is what also came to my mind when first read your post, Matt seems to be in the same inclination.

And my bet is you will have to build a thin _transport_ wrapper protocol to the _call_ protocol as was done with HTTP ...

If I'm not completely off, please consider taking this out to a formal document and post here so it can be discussed / published properly.

Regards,
Lo.

--
You received this message because you are subscribed to the Google Groups "JSON-RPC" group.
To view this discussion on the web visit https://groups.google.com/d/msg/json-rpc/-/JFuwUzSOQWcJ.
To post to this group, send email to json...@googlegroups.com.
To unsubscribe from this group, send email to json-rpc+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/json-rpc?hl=en.

[Matt (MPCM)]

To your other point, the client would end up handling this in the same way as a broken/unresponsive server.

Especially with aync, at some point you have to settle the accounting on the client side and determine which calls are still outstanding, which one's you want to consider 'lost', etc.

On Tuesday, October 9, 2012 2:34:07 PM UTC-4, David Lee wrote:

[Roland Koebler]

Hi,

> But what then? Since TCP is a streaming protocol, how would
> the server resync back with the client and start processing requests again?

you *always* need some way to tell the server that the request/transmittion
is complete and some error handling/error recovery. I know several ways:

1. One connection per transmission:
Use a separate connection for each request/response-pair, and shutdown
the writing side of the socket after the data is sent. Use a timeout
to remove old connections with broken/truncated/stalled transmissions.

This is quite simple and straigt forward, and works well.
My own JSON-RPC library works this way.

2. Send several requests/responses over a single TCP connection:

If you really need to send several requests over a single TCP
connection (e.g. if you *really* need the performance), you could
use a TCP connection for several JSON-RPC transmissions. But then,
things get a bit more complicated:

a) detect the end of a JSON-RPC transmission
You have to detect when a JSON-object is complete, by e.g.:

- using a streaming JSON-decoder, which detects when a JSON-object
is complete; unfortunately, most JSON-decoders cannot do this.
- using a "JSON-splitter", which detects the end of a JSON-object
(I wrote one in Python in about 50 LOC)
- using some small self-delimiting encoding on top of TCP; I would
prefer Netstrings (http://en.wikipedia.org/wiki/Netstring).

b) error-recovery
You have to detect incomplete/broken/truncated JSON-objects, and
re-sync to the next valid JSON-object.

In my opinion, this should be done by timeout and re-connect:
If a JSON-RPC-transfer does not complete in a specific time, send
back all outstanding responses + a parse error, then close the
connection and wait for the client to reconnect.

I would recommend to use one connection per transmission (1).
If you think that this costs too much performance for your application,
then please read about the advantages and disadvantages of persistent
connections (e.g. http://en.wikipedia.org/wiki/HTTP_persistent_connection),
before chosing (2).

> Similarly, how would it handle a truncated message?
>
> --> {"jsonrpc": "2.0", "method": "foo", "params": [1, 2,

Respond with a parse error (after a timeout) and close the connection.


regards,
Roland

[David Lee]

On Tuesday, October 9, 2012 8:10:43 PM UTC-5, Matt (MPCM) wrote:

Hi,

The server's job is to accept strings, break them into chunks via something like (End Of Transmission) 0x04, parse those into valid json objects and then into valid json-rpc requests to be handled. So... unless something like EOT is sent (there are old posts discussing separator/EOT approaches), the question then becomes how large of a buffer will server use before it throws an error filled response object.

Sure, if there's some reliable mechanism to detect the end of a message, then error recovery isn't a problem at all.

*looks for EOT in the old posts*

If how messages are transported isn't specified, then how does interoperability work? I can't see how it could work if the client and server pick different mechanisms for transporting messages.

I've looked at few JSON-RPC libraries, and none of them seem to specify how they transport messages over TCP. How can I know that I can use jsonrpc4j on the server side and json-rpc-perl6 on the client side and they'll work together?

If the server is doing partial object building as the string goes along, it could know early on that the parse will fail.... Even then, it needs to be able to junk any additional input until it can make a clean start. At some point the server will need to give up, without a clean truncation point.

If you are dealing one stream per client/server connection, would it be valid approach to terminate the connection?

Hopefully others using/writing streaming clients more regularly than me can comment on their approaches. Generally this is only happening when the json is not being generated by an encoder directly, perhaps a client spewing bad data?

--
Matt (MPCM)

 

-- 

dave

<><

[David Lee]

On Wednesday, October 10, 2012 8:28:00 AM UTC-5, . wrote:

1. One connection per transmission: 

   Use a separate connection for each request/response-pair, and shutdown
   the writing side of the socket after the data is sent. Use a timeout
   to remove old connections with broken/truncated/stalled transmissions.

   This is quite simple and straigt forward, and works well.
   My own JSON-RPC library works this way.

This would be problematic because my server needs to send asynchronous notifications back to the client. That requires a persistent connection for the server to have a channel to send the notification back on.

It occurs to me, though, that this is something else that isn't in the spec. It's allowed, but not required.

Can anyone speak as to whether most JSON-RPC implementations can fill both client and server roles, at the same time, to the same client?

regards,
Roland

Thanks!

-- 

dave

<>< 

[Matt (MPCM)]

On Wednesday, October 10, 2012 10:54:15 AM UTC-4, David Lee wrote:

On Tuesday, October 9, 2012 8:10:43 PM UTC-5, Matt (MPCM) wrote:

Hi,

The server's job is to accept strings, break them into chunks via something like (End Of Transmission) 0x04, parse those into valid json objects and then into valid json-rpc requests to be handled. So... unless something like EOT is sent (there are old posts discussing separator/EOT approaches), the question then becomes how large of a buffer will server use before it throws an error filled response object.

Sure, if there's some reliable mechanism to detect the end of a message, then error recovery isn't a problem at all.

*looks for EOT in the old posts*

If how messages are transported isn't specified, then how does interoperability work? I can't see how it could work if the client and server pick different mechanisms for transporting messages.

The 2.0 spec was made specifically transport agnostic for this reason. Declaring how a certain transport(s) must be used is always going to fall short, as there are simply too many preferences to be considered. You can make certain assumptions about the client and server understanding each other at the data level and how each will react, but it is the role of implementors to get the data from client to server. I like to see implementations that implement the core client/server in the language, then so provide extensions of that core for various transports.

 

I've looked at few JSON-RPC libraries, and none of them seem to specify how they transport messages over TCP. How can I know that I can use jsonrpc4j on the server side and json-rpc-perl6 on the client side and they'll work together?

You can't know... but what you would know is that messages from the client will be understood by the server once it arrives. If the server is written for TCP streams and the client only uses http, that gap isn't going to be bridged by the specification. TCP to TCP is also not addressed for the same reason. Best practices, or at a minimum common practices, do arise and tend to be followed. Generally, if you are using the transport aspect in an uncommon way, you are best served using a matching pair of client/server libraries that agree on the transport mechanism and details.

[Matt (MPCM)]

On Wednesday, October 10, 2012 11:23:45 AM UTC-4, David Lee wrote:

On Wednesday, October 10, 2012 8:28:00 AM UTC-5, . wrote:

1. One connection per transmission: 

   Use a separate connection for each request/response-pair, and shutdown
   the writing side of the socket after the data is sent. Use a timeout
   to remove old connections with broken/truncated/stalled transmissions.

   This is quite simple and straigt forward, and works well.
   My own JSON-RPC library works this way.

This would be problematic because my server needs to send asynchronous notifications back to the client. That requires a persistent connection for the server to have a channel to send the notification back on.

A json-rpc server cannot send notifications... that is something reserved for the client. If your running a pair of client/servers on each end of the transport, this can be done of course. 

It occurs to me, though, that this is something else that isn't in the spec. It's allowed, but not required.

A client is always a producer of request objects and a consumer of response objects.
A server is always a consumer of request objects and a producer of response objects.

Transport(s) connect peers who share objects.

 

Can anyone speak as to whether most JSON-RPC implementations can fill both client and server roles, at the same time, to the same client?

Some fill both, but generally not at the same time and not over the same transport connection. I don't think most people use the bidirectional nature that the specification allows. There will be people on this list who use it in this way, but for the tech world in general I would say that is not how it is widely used this way.

I've played around with bidirectional notification only systems, sort of like udp peering... but in general people reason about their challenge at hand and use json-rpc in an implied sequential style... this has been my observation thus far.

 

regards,
Roland

Thanks!

-- 

dave

<>< 

--
Matt (MPCM) 

[Henry Unger]

Since when is it prohibited for a server to send a notification to a client? I don't recall reading that in the original spec. I've been doing just that for years.

Best regards,

Henry

--

You received this message because you are subscribed to the Google Groups "JSON-RPC" group.

To view this discussion on the web visit https://groups.google.com/d/msg/json-rpc/-/b0mwLzsZoOMJ.

[Shane Green]

That depends on the details.  Assuming HTTP/S medium… 

A JSON-RPC Service accepts JSON-RPC requests and returns JSON-RPC responses.  The term "service" should be thought of as a role, more than anything else: the service/server role accepts requests, and returns responses; it does not initiate requests and receive responses.  It may happen that the same application is also playing the role of a JSON-RPC client, in which case it is initiating requests sent to a JSON-RPC service, and receiving that service's responses.  That service may or may not the same end point that is also a client.  If it is the same end point, then you have bidirectional JSON-RPC because both applications are operating in both roles in relation to one another: A is the server for client B, and B is a server for client A.  If you were to pipe this bidirectional behaviour through a single connection then you would have an odd setup, as you'd have requests being initiated in each direction over the same pipe.  This would not be impossible, or even all that challenging, but it probably wouldn't be a huge benefit unless you were attempting to run two way communication through a network that only allowed one of your two end point to b an actual server and receive incoming connections on port 80, for example.  Then you might have the client connect to port 80, then hijack the channel for the server–acting as a client–to initiate requests to the client (acting as a server).  You would have to have serious control over the web-server to do this, though. 

Shane Green 

www.umbrellacode.com

805-452-9666 | sh...@umbrellacode.com

[Shane Green]

I think the reason 99% of JSON-RPC conversations are unidirectional (hehehe) is probably because, even though the JSON-RPC specification doesn't specify it, 99% of its usage is over HTTP/S.  It's quite uncommon, in my experience, to have bidirectional HTTP/S going on.  The second you start using JSON-RPC over something like a simple socket connection, its bidirectional and asynchronous capabilities immediately become topics of consideration.  

Shane Green 

www.umbrellacode.com

805-452-9666 | sh...@umbrellacode.com

On Oct 28, 2012, at 7:53 PM, Henry Unger <hun...@gmail.com> wrote:

[Matt (MPCM)]

On Sunday, October 28, 2012 10:54:06 PM UTC-4, Henry wrote:
> Since when is it prohibited for a server to send a notification to a client? I don't recall reading that in the original spec. I've been doing just that for years.

Since 2.0 defined the server and client roles in more detail. 1.0 was vague on some of these points, as it didn't forbid them and the language/examples left things open to interpretation.

2.0 allows this, but it technically the end point is acting as both a client and a server role. In bidirectional setups this is usually the case.

[Shane Green]

You could create a solid asynchronous invocation model by having a invocation manager on the client that creates (or gets) a JSON-RPC request object, creates something like a deferred and stores it in a dictionary by that request ID, enqueue the request for sending to the server, then returns the deferred to the caller.  On the server side anytime the request handler got a deferred back from invoking a local method, or perhaps anytime it invoked a method regardless of the return value, it registered completion of the response as a callback on the deferred and carried on handling requests.  Each time the request manager receives a response back on the client, it grabs the request ID, looks up the corresponding deferred, and completes it with the response.  Normally HTTP responses are assigned to corresponding requests by sequence, but this removes that restriction, so requests and responses can be sent in any order, and opens the door to long running invocations without blocking subsequent requests, binding up server threads, or requiring separate channels. It could therefore be used to create a very event-driven like collaboration, wherein the client always maintained a request for some event, and the server returned that event if and when it happened.  The only thing that needs to be dealt with is the fact that a connection can be closed at any time… I think.

Shane Green 

www.umbrellacode.com

805-452-9666 | sh...@umbrellacode.com