Calling from Python to JSON-RPC server in Go

[Bruce Eckel]

Has anyone had any luck with this?

I have a super-simple Go server and client that communicate with JSON-RPC (the server just echoes the argument you send it).

I've tried several Python JSON-RPC libraries and the ones I can get running all produce the same result when I try to call the Go Server:

rpc: rpc: server cannot decode request: invalid character 'P' looking for beginning of value

It doesn't matter what call I'm making, including "listMethods," I get the same result (and it's the server printing out the error message; I know because I have it running in a separate window).

Actually, has anyone communicated with a Go JSON-RPC server using any language other than Go?

Thanks for any help you can give, or suggestions for attacking the problem. I can attach the code if that will help.

-- Bruce Eckel
www.Reinventing-Business.com
www.MindviewInc.com

[Brian Ketelsen]

On Aug 21, 2011, at 9:23 PM, Bruce Eckel wrote:

>
> Actually, has anyone communicated with a Go JSON-RPC server using any language other than Go?
>

A few months ago I had good success with the built-in JSON-RPC server in Go and a Ruby client.

Brian

[Stephen Day]

Unfortunately, most python json-rpc implementations don't seem to separate the http and json-rpc protocol, which ties each rpc call to a separate http request. Go actually uses a CONNECT call to upgrade to a regular socket connection, then passes the calls back and forth over that. Getting python to work with this isn't hard; there is just nothing off the shelf.

-steve

[Bruce Eckel]

If that's an efficiency issue then it doesn't impact me -- I'm just going for a proof of concept. I would have used XMLRPC but Go has the built-in JSONRPC.

Are you saying that the JSONRPC libraries for Python don't communicate with Go's JSONRPC? (If so, I ask: "How is this a standard?").

It's very possible I'm doing something basic wrong. I think perhaps I need to clean up the examples a little and do a couple more experiments, then post them so someone can tell me if there's some obvious problem.

-- Bruce Eckel
www.Reinventing-Business.com
www.MindviewInc.com

[Russ Cox]

On Mon, Aug 22, 2011 at 18:44, Bruce Eckel <bruce...@gmail.com> wrote:
> If that's an efficiency issue then it doesn't impact me -- I'm just going
> for a proof of concept. I would have used XMLRPC but Go has the built-in
> JSONRPC.
> Are you saying that the JSONRPC libraries for Python don't communicate with
> Go's JSONRPC?

Yes.

The jsonrpc package is doing jsonrpc over raw TCP.
The Python server is expecting it as the body exchange
of an HTTP request/response on a particular URL.

Please do file a bug: it should be easy to add support
for URL endpoints.

> (If so, I ask: "How is this a standard?").

It's not a standard. It's a web page some
guy wrote, and it's not terribly fleshed out.
http://json-rpc.org/wiki/specification

Go implements 2.1. Python implements 2.2.
This sentence from 2.2 is particularly relevant:
"FIXME: Describe/define how sessions are handled,
as there is no implicit session formed by a TCP-connection."

Russ

[Stephen Day]

I put together a minimal python client/go server example in this GIST:

https://gist.github.com/1164621

Its not fancy but I hope this helps.

-steve

[Bruce Eckel]

Exceptionally helpful, as in: it works!

So if I understand correctly, all the existing JSONRPC Python clients are using http which the Go library doesn't have support for yet (thus a bug report is needed), and your example works because it is going directly through a socket.

Thanks VERY much -- this will tide me over in the meantime. I don't think I quite have enough understanding to submit a bug report yet, but I'll get there.

-- Bruce Eckel
www.Reinventing-Business.com
www.MindviewInc.com

[Sugu Sougoumarane]

There is a new function in rpc (ServeRequest) that can be used to trivially implement a pure HTTP protocol. Here's what you need to do (I paraphrased this code out of a bigger framework):

// Name package

// import as needed

// ServeHTTP handles json rpc requests in HTTP compliant POST form

func ServeHTTP(codecName string, cFactory ServerCodecFactory) {

  http.HandleFunc("/_json_http_", ServeHTTP)

}

func ServeHTTP(c http.ResponseWriter, req *http.Request) {

  conn := &httpConnectionBroker{c, req.Body}

  codec := jsonrpc.NewServerCodec(conn)

  rpc.ServeRequest(codec)  // returns os.Error, handle if needed

}

// Emulate a read/write connection for the server codec

type httpConnectionBroker struct {

  http.ResponseWriter

  io.Reader

}

func (self *httpConnectionBroker) Close() os.Error {

  return nil 

}

[Bruce Eckel]

I'm starting from significant ignorance here (maybe this is already how things work), but would it make sense for the RPC implementation to try to use sockets if it's a local connection, but go to http if it's over the internet? I actually don't even know whether that would make a performance difference.

The example I'm trying to create is to use Python for doing some things, and then to put a parallel computing portion onto a Go server and solve that part of the problem using Go's parallelism. It's an example of the benefits of polyglot programming within a single solution. What Stephen sent will get it working, but eventually I'd like to see simpler code because it makes a better case to have an easy RPC call.

Thanks for all the help so far.

-- Bruce Eckel
www.Reinventing-Business.com
www.MindviewInc.com

[Skip Tavakkolian]

On Tue, Aug 23, 2011 at 9:12 AM, Bruce Eckel <bruce...@gmail.com> wrote:
> I'm starting from significant ignorance here (maybe this is already how
> things work), but would it make sense for the RPC implementation to try to
> use sockets if it's a local connection, but go to http if it's over the
> internet? I actually don't even know whether that would make a performance
> difference.

given that it is RPC, i don't see how any of the HTTP speed tricks are
going to be useful.

[Bruce Eckel]

OK, good to know. Then at some point the only thing we need to do is figure out what the bug is and fix that.

-- Bruce Eckel
www.Reinventing-Business.com
www.MindviewInc.com

[Stephen Day]

Bruce:

I think there are two issues that you are looking at:

1. Python doesn't have an rpc implementation (of which I am aware) that uses a CONNECT method to upgrade an http connection to a bare socket. Perhaps the Go rpc implementation, which is considerably well thought out, can influence the currently available or future python rpc implementations.

2. Go's support for the more prevalent rpc implementation, as described by Sugu, is not obvious from looking at the package documentation.

If there is some work to resolve these two issues, from both sides, I think python/go can be an excellent pairing in a multi-language environment.

 

 What Stephen sent will get it working, but eventually I'd like to see simpler code because it makes a better case to have an easy RPC call.

Extrapolating what I sent into a full bells and whistle RPC implementation is not that far off. I've updated the GIST, with an example:

https://gist.github.com/1164621

This doesn't really have what I would call sophisticated error handling, but it does abstract the calls to some extent. 

The point I am trying to get at is that you don't really need to involve HTTP in jsonrpc. Its nice to use, in that it acts as a proxy and you can decorate it with http features. All that needs to be done to get python working with the http-based go rpc is to issue a CONNECT request on an http connection in python.

As an aside, I may have indeed found a bug. If you run the server from the gist, then run this in the shell, the server goes into an infinite loop:

echo '{id:1}' | nc 127.0.0.1 5090

The problems seem to be mishandling of malformed informed input on a codec.

-steve

[Bruce Eckel]

Outstanding! And I think I should be able to get it working in Python 3 without much trouble.

Small detail: I *think* you can replace:

if resp.get('error') is not None:

with simply:

if resp.get('error'):

(Although your way is more Go-ish, I suspect).

When I was trying the various Python JSONRPC libraries I was getting an infinite loop on the server, so you might have found the same bug.

Anyway, thanks very much and this looks like it neatly solves my problems.

-- Bruce Eckel
www.Reinventing-Business.com
www.MindviewInc.com

[Bruce Eckel]

I thrashed around with the code to make it work on both Python 2 and 3, and turned it into a simple module. Are there any issues using/publishing the following?

# jsonclient.py

# A simple JSONRPC client library, created to work with Go servers

# Written by Stephen Day

# Modified by Bruce Eckel to work with both Python 2 & 3

import json, socket, itertools

class JSONClient(object):

    def __init__(self, addr, codec=json):

        self._socket = socket.create_connection(addr)

        self._id_iter = itertools.count()

        self._codec = codec

    def _message(self, name, *params):

        return dict(id=next(self._id_iter),

                    params=list(params),

                    method=name)

    def call(self, name, *params):

        request = self._message(name, *params)

        id = request.get('id')

        msg = self._codec.dumps(request)

        self._socket.sendall(msg.encode())

        # This will actually have to loop if resp is bigger

        response = self._socket.recv(4096)

        response = self._codec.loads(response.decode())

        if response.get('id') != id:

            raise Exception("expected id=%s, received id=%s: %s"

                            %(id, response.get('id'), 

                              response.get('error')))

        if response.get('error') is not None:

            raise Exception(response.get('error'))

        return response.get('result')

    def close(self):

        self._socket.close()

-- Bruce Eckel
www.Reinventing-Business.com
www.MindviewInc.com

[Stephen Day]

I filed an issue on the malformed input problem described earlier:

http://code.google.com/p/go/issues/detail?id=2179

> Small detail: I *think* you can replace:

> if resp.get('error') is not None:

> with simply:

> if resp.get('error'):

> (Although your way is more Go-ish, I suspect).

Yes, you can, but the latter effectively wraps the expression with bool(), which may force a call to __bool__ on the result of resp.get('error'). The "is" keyword just does a simple identity check against None, which avoids possibly expensive function call overhead or some notion of truth that is unintended.

> I thrashed around with the code to make it work on both Python 2 and 3, and turned it into a simple module. Are there any issues using/publishing the following?

Feel free to use or publish it under the MIT license, ensuring I'm credited. If you need this to be more formal, please contact me directly.

Cheers,

Steve