app with net/rpc over websocket - some design problems

[Steve OConnor]

Just want to run this design problem that I have at my end past you fellow gophers.

Its a little bit complicated, so please bear with me if the description is a little verbose .... its important to give all the details up front in this case, even if that does make for boring reading.

Your opinions will be most welcome !

Background Info & Glossary :

Golang's net/rpc package allows a Go program to make function calls against another (remote) Go program.

The function call is in the form :    rpcClient.Call(nameOfRemoteFn string, params Type, returnedValues Type)

The rpc calls work fine over a number of different connection types, including websockets

The rpc calls use Go's own encoding/gob binary format by default (it does support custom formats as well, but that is hard work)

GopherJS is a Go -> JS transpiler that allows you to write browser apps in Go, and it supports :

- full range of Go types

- goroutines

- channels

- rpc, with gob encoding

- DOM manipulation

- 3rd party JS libs

Situation :

- Multi User game style of application, with a requirement for a constant volume of bi-directional data flow between FrontEnd and BackEnd

- Standard "web application" architecture, with a Go server at the backend, which handles standard HTTP requests with no problems.

- Standard SPA "web application" at the front end, also written in Go  (GopherJS)

- On startup, the front end loads itself up and uses vanilla HTTP protocols to communicate with the backend. This includes loading resources, images, and basic navigation through some publicly available pages.  So far so good.

- When the user at the front end decides to Login to the guts of the system, the following steps happen :

1. FrontEnd opens a new websocket connection to the BackEnd, and then registers itself as an rpc client on that connection
   
2. BackEnd receives new websocket, and then registers a "Login" service on that websocket using net/rpc
   
3. FrontEnd renders a login form. User enters login credentials, then clicks "submit"
   
4. On Login submit, FrontEnd makes a call to the "Login" service via rpc, using the login creds entered on that form
   
5. ....   <binary frames to and fro with gob encoding magic here> ....
6. On fail, the socket is dropped, and the user has to try again.  GOTO 1
7. On successful login - FrontEnd receives a whole lot of data in the form of a Go struct, and then gets on with it

At this stage, the BackEnd knows which user is associated with that socket connection, so it can treat all data over that connection as validated, and can easily associate other application level context with that socket from here on.

BackEnd then registers any other functions needed for that user as rpc services on that socket.  Note that depending on who it is that logs in, the set of functions can be very different in each case, and this (as it turns out) is an exceptionally elegant way of managing access to the BackEnd functions on a per-user basis. (ie - no more need for cookies, JWT tokens, decryption, per-message validation of REST requests, blah blah blah == very fast and secure comms with minimal code) 

At this stage, the FrontEnd now has all the "world state data" sitting in memory, in the form of Go Types, and a whole set of Go functions that it can call directly on the BackEnd. Very nice !

Numerous Problems :

1) As things happen in the "game world", the server will need to send updated world-state data to each connected client.

Such updates from the BackEnd to the FrontEnd do not need to be replied to .... so they can be simple messages (or complete RPC calls to the FrontEnd ... doesnt really matter)

So the FrontEnd now needs to look for random messages from the BackEnd, and act on them when they happen, without interfering with the outgoing RPC calls

This is a classic Client/Server problem, just raising its head once again, in a new context.

Looking at the RPC over the wire, it goes roughly like this :

1. Client sends some binary frames over the connection
2. <some delay>
3. Server sends some binary frames over the same connection in reply

... so any additional data sent over the same connection between steps 1 & 3 will break things

2)  Network timeouts and Keepalives

In order to keep the socket alive, (in general, and when used through Nginx or similar) Pings and KeepAlive packets may be needed.

If so, keepalive packets will probably be generated from a separate goroutine that has no knowledge of the state of any other RPC calls that may be in transit

Whatever keepalive strategy is used, it cant interfere with the RPC calls that might be in transit

3)  Loss of connection

If the connection drops out, the FrontEnd will (automatically) re-connect, and the application state will be restored.   I have that covered, but thought I should mention it.

Numerous Possible Solutions  :

1. Write a new layer of middleware that sits between the websocket and net/rpc, to magically re-order any  network frames in the case of some intermediate message corrupting the RPC call ?  
2. Dont use net/rpc in this case, fall back to an async message passing protocol. That would work ..... but still, I would <like> to use rpc for this if I can, because coding should be about fun and discovery !
3. Open 2 sockets - one that is strictly RPC to the BackEnd, and another socket that is just for world state updates to the FrontEnd.     .... That would work too, but sounds ugly

What do you gophers think ?

Am I missing the obvious here ?

[jasonan...@gmail.com]

In my implementation of RPC over websocket I take advantage of the websocket message type bits to indicate to each side what the payload is. The RPC data uses one message type and other communication uses a separate message type. 

Both github.com/gorilla/websocket and golang.org/x/net/websocket has good support for various message types.

[Vibhav Pant]

On Saturday, January 30, 2016 at 1:27:06 PM UTC+5:30, Steve OConnor wrote:

1. Open 2 sockets - one that is strictly RPC to the BackEnd, and another socket that is just for world state updates to the FrontEnd.     .... That would work too, but sounds ugly

It should be possible to use the same WebSocket connection for those two tasks, but you'd need to provide your own ClientCodec implementation to prevent concurrent read and writes to the underlying WS connection.

[Steve OConnor]

It should be possible to use the same WebSocket connection for those two tasks, but you'd need to provide your own ClientCodec implementation to prevent concurrent read and writes to the underlying WS connection.

Thanks Vibhav, I do believe that is on the right track for a solution.  Will investigate further when I get some "studio time"

I skimmed over the documentation on that part before, and didnt give it a 2nd look, so I didnt really grok what that function was trying to achieve. It looks like it is used for creating custom encodings beyond vanilla gob encoding ?

However, going through the source code for the RPC client    https://golang.org/src/net/rpc/client.go   ... it looks like the internals make use of a couple of private mutexes to control access to the wire, and use a seq scheme to possibly avoid this entire family of issues. The same mutex dance is mirrored in the RPC server code.   

Might be able to make this fully bi-directional without too much work  (if its not already doing that by default). Will run some experiments later and try to break it.

Worst case would be to write a simple app level codec that still used gob, but also used an extra app-visible mutex, then I reckon that would allow running both a client and a server at the same time, over the same connection. Looking forward to some play time to try that out.

cheers

 

[Steve OConnor]

Bit of an update on this one  ... seems that I missed the obvious after all

I was assuming that when you create an rpcClient over a websocket, it did the following :

0 - Do nothing until told to send a message

1 - Send a Request to the server

2 - Read a Response header

3 - Read a Response body

I was looking for a way to safely use the connection in between outgoing RPC calls (state 0 in the above list), possibly using channels or mutexes even to control access to the connection.

The reality is a little different.  As soon as you create a new rpcClient, it effectively creates a "server" on the Client end, and kicks off a read loop on the underlying connection looking for RPC Response headers on the wire. This is before the first request goes out.

So that means that the rpcClient will process "responses" from the server, even if it hasn`t issued a request yet.  This means that the server can send unsolicited messages to the Client, and they will be processed by the rpcClient code, as if in response to an actual request. Provided that the messages obey the RPC protocol, and you send valid headers that is.

A very minor bit of code for a custom ClientCodec  to look at incoming headers and branch off to the correct code to handle any async messages from the server looks like it will do the trick.  

I haven't quite got this working yet, and my code is a mess, but I will post up some examples once I get it sorted.