Re: [tornado] a few sockjs-tornado questions

[Serge S. Koval]

Hi,

On Sat, Nov 3, 2012 at 4:59 AM, funback <expe...@gmail.com> wrote:

hi, just discovered sockjs-tornado by recommendation, looks very nice and would like to use it more seriously. But first, a few questions I'd like to clarify (some may seem obvious for a current user of the library).

1. is there a easy way to measure how much time it takes to handle a request to a sockjs-tornado method (e.g. on_message)? In tornado, it gives you the time in milliseconds for handling each request, can this be easily done for sockjs-tornado methods?

It is quite complicated, as there's no usual request-reply pattern and sockjs-tornado can't figure out if you finished processing incoming message or just started asynchronous operation.

If you don't use asynchronous code or use @gen.engine, you can measure execution time by getting time stamp in beginning of on_message and calculating delta in end of the function.

Also, take a look at http://sockjs-tornado.readthedocs.org/en/latest/stats.html - it has some basic counters. There's also example how to access these. If you have need for more counters and everyone will benefit from them, open a ticket: https://github.com/mrjoes/sockjs-tornado/issues?state=open

 

2. i suspect the send/broadcast method in sockjs-tornado is implemented asynchronously (is it?), then how should i handle this situation: imagine i'm broadcasting a set of 5 clients, and when sending the msg to the 3rd one, some exception/error happens (for example, this client may have already disconnected), now what will happen to the 4th and 5th clients? Will the msg be sent to them at all?

Yes, message will be sent to all of them, except of faulty client. In most of the cases, exception won't happen in the send() call, instead it will be raised asynchronously, when Tornado receives status of the asynchronous operation. However, for rare cases when send() fails, every sockjs-tornado transport has safeguard, like here: https://github.com/mrjoes/sockjs-tornado/blob/master/sockjs/tornado/transports/xhr.py#L40

 

3. i notice that there is is_closed method on the SockJSConnection class. Can this method be used to detect disconnections in real-time, while executing on_message of some connection instance?

This is just a flag and it only represents if logical session was closed. In most of the cases it also represents if connection was closed, but if client is disconnected while your code was running (ioloop was blocked), it will report wrong value.

 

4. most importantly, how should I handle connection-related exceptions in on_message method? For examples, network errors may occur when trying to send msg; connections may already dropped when sending msgs asynchronously; how do we handle these exceptions/errors?

I'm not sure if you have to do something special about it. If your code raises exception, sockjs-tornado will kill offending connection and will continue working like nothing happened.

So, if you expect your code might raise exception, wrap it in try/except or ExceptionStackContext and handle errors yourself. 

If you see any kind of unexpected behavior, open a ticket.

Serge.

[funback]

hi serge, thanks for the answers they are very useful. two further questions:

1. you mentioned the use of @gen.engine decorator, do you apply this directly on on_message method? 

2. you mentioned 'If your code raises exception, sockjs-tornado will kill offending connection and will continue working like nothing happened', so how does this apply to the broadcast example:

Again, a set of 5 clients, and client #1 calls the broadcast() method, which fails on client #3. now you've confirmed that the msg will be sent to all clients #1, #2, #4 and #5, but will client #1 be affected here (as in be treated as a offending connection and killed by sockjs-tornado) because it calls the offending method?

cheers.

[Serge S. Koval]

Hi,

On Sat, Nov 3, 2012 at 2:28 PM, funback <expe...@gmail.com> wrote:

hi serge, thanks for the answers they are very useful. two further questions:

1. you mentioned the use of @gen.engine decorator, do you apply this directly on on_message method? 

You can apply it to any function if you need to make asynchronous calls in sort-of-synchronous manner. This includes on_message.

 

2. you mentioned 'If your code raises exception, sockjs-tornado will kill offending connection and will continue working like nothing happened', so how does this apply to the broadcast example:

I see what you mean. No, send() won't raise exception if IOError happens when it is trying to send some data to the client. So, client #1 won't be killed if one of the clients disconnected during broadcast(). However, if something else will raise exception - yes, client #1 will be closed, as code is running in context of client #1 context.

Serge.

[funback]

Hi serge,

just a couple of new questions as I am getting started with sockjs-tornado.

1. I saw this thread: https://groups.google.com/forum/?fromgroups=#!topic/sockjs/drG5B4PhmeE, where you mentioned there is ping/pong available for sockjs. How is this implemented, in other words, how do I send a ping and receive a pong in sockjs-tornado? In addition, if this facility is available, does it work for all transport (not just websocket)?

2. how do i enable ssl for sockjs-tornado? do i just hook it up with a url which uses ssl scheme?

thanks for your help.

[Serge S. Koval]

Hi,

 SockJS does not have ping/pong yet - it is in development branch and not yet released.

 

 If you need ping/pong, you will have to implement it yourself: send ping message to the client and catch all responses. If there were more than X responses missed, close connection.

 Two options:

1. Terminate SSL at load balancer (new HAProxy can do it), so Tornado will see ordinary HTTP requests;

2. Use built-in Tornado SSL stuff. There's example for TornadIO2, which I did not port to the sockjs-tornado: https://github.com/mrjoes/tornadio2/blob/master/examples/ssl_transports/ssl_transports.py#L68

Serge.

[funback]

hi serge,

related to ping/pong, is it possible that a connection remains open between client and server, but a msg sent (by either server or the client) goes missing during the transmission? Or is it always the case that if anything goes wrong when sending a msg (from either end), the connection will ALWAYS close?!

I will try haproxy for the ssl, thanks for the suggestion.

[Serge S. Koval]

Hi,

 If you sent a message, there is no guarantee it was delivered. It is quite possible that something bad happened with users network and TCP connetion will timeout in few minutes, while your application still think that user is online.

 So, if you want to be absolutely sure that message was delivered, you need to invent some kind of message acknowledgement mechanism, each message should have its own ID, etc. With pings you'll know that user died sooner than TCP stack, especially with aggressive timeout values. But it won't tell you which messages user received before he went offline.

Serge.

[funback]

hi serge, so a connection remains open does NOT guarantee every msg sent between client and server is delivered? in that case, will the following work:

1. whenever an endpoint (server or client) sends a msg, it also caches the msg in case it needs to re-send.

2. after sending the msg, it expects an ACK reply to confirm the the msg is delivered. if no such confirmation after some timeout, re-sends the msg from cache.

3. at the recipient end, store the incoming msg in a cache so that it can compare with the next msg in case of duplicates, so we don't process the same msg twice. duplicates may happen when the ACK reply gets lost, and the other end re-sends the msg.

If the above works, it solves half of the problem - guarantee msg delivery while the sockjs connection remains open.

The other half of problem is disconnection. In my case, i can ignore the case where client or server crashes (i.e. browser has to be re-opened or server has to restarted). But if the client web page remains open, then I want to restore the working state of client and server after re-connection. All computation on client and on server in my case is triggered by msgs. For example, if the client receive a msg saying: 'what is 2 + 2', then client performs the calculation. Without the msg, the client (or the server) stay idle. After re-connection, the client (or the server) might not know what to do next, as it needs to figure out whether it should send a msg or expect to receive one. Any suggestion how to make this work, or some reference i can look to?

thanks. 

[Serge S. Koval]

Hi,

 My comments are inline.

On Tue, Nov 6, 2012 at 6:42 PM, funback <expe...@gmail.com> wrote:

hi serge, so a connection remains open does NOT guarantee every msg sent between client and server is delivered?

SockJS works on top of TCP protocol. Everything that is applicable to TCP is applicable to SockJS. If TCP connection is open, ACK packets going back and forth, it is guaranteed that data you send will be delivered in order.

If remote endpoint dies (switch died, ISP screwed up, packets are no longer routable, etc), TCP connection will be considered open for some more time, usually few minutes. Even though you attempted to send something through this connection, it will be buffered by your OS TCP stack and you won't get application-level error right away. After timeout kicks in, socket will be closed and you will lose all data in the socket buffer.

  

in that case, will the following work:

1. whenever an endpoint (server or client) sends a msg, it also caches the msg in case it needs to re-send.

2. after sending the msg, it expects an ACK reply to confirm the the msg is delivered. if no such confirmation after some timeout, re-sends the msg from cache.

3. at the recipient end, store the incoming msg in a cache so that it can compare with the next msg in case of duplicates, so we don't process the same msg twice. duplicates may happen when the ACK reply gets lost, and the other end re-sends the msg.

This approach will work, but it is somewhat wasteful in terms of latency, traffic and resources. 

What are you trying to achieve: guarantee that every single message was received by the client in time (thus ack for every message) or just restore client state after reconnection in case of network problems? If it is just restoring state:

1. Assign each message its own unique ID, queue it on the server-side and send it to the client;

2. Whenever client has something to send or certain amount of time passed, send last received ID to the server, so it has chance to cleanup outgoing queue;

3. Upon reconnection, send last received ID and server will send back any missed messages.

In this case, you're still confirming all messages received, but you don't care when message was received (before reconnection, after reconnection, etc). For this to implement, you will need some kind of session storage, which is disconnected from sockjs-tornado connections.

Also, in most of the cases you can treat reconnection as initial connection. It might require to clean some internal data structured though.

Serge.

[funback]

hi serge, please let me explain what i am thinking to do.

What are you trying to achieve: guarantee that every single message was received by the client in time (thus ack for every message) or just restore client state after reconnection in case of network problems? 

my case needs guarantee for both ways, because:

1. if a msg from client to server is lost, the server just waits for msg and hangs there.

2. if a msg from server to client is lost, the client just waits for msg and hangs.

in other words, without receiving msg (and the correct one), both server and client can do nothing but wait. Both ends rely on these msgs to function correctly.

If i understand you correctly, whenever a msg goes missing it will always result disconnection in the end. so really, i need a mechanism for both server and client to re-send any msg that fails to deliver, so that both server and client can resume their respective operation. 

If it is just restoring state:
1. Assign each message its own unique ID, queue it on the server-side and send it to the client;
2. Whenever client has something to send or certain amount of time passed, send last received ID to the server, so it has chance to cleanup outgoing queue;
3. Upon reconnection, send last received ID and server will send back any missed messages.
In this case, you're still confirming all messages received, but you don't care when message was received (before reconnection, after reconnection, etc). For this to implement, you will need some kind of session storage, which is disconnected from sockjs-tornado connections.

considering my analysis above, this approach only works for ensuring msg delivery from server to client, but not the other way around. do correct me if i misunderstood this approach above.

would like to hear your idea here. thanks for listening.

[Serge S. Koval]

Hi,

On Tue, Nov 6, 2012 at 11:40 PM, funback <expe...@gmail.com> wrote:

in other words, without receiving msg (and the correct one), both server and client can do nothing but wait. Both ends rely on these msgs to function correctly.

If i understand you correctly, whenever a msg goes missing it will always result disconnection in the end. so really, i need a mechanism for both server and client to re-send any msg that fails to deliver, so that both server and client can resume their respective operation. 

Yes, that's correct.

 

considering my analysis above, this approach only works for ensuring msg delivery from server to client, but not the other way around. do correct me if i misunderstood this approach above.

You're right - server won't know it missed some messages from the client. You can count messages from both ends and exchange IDs if you need to keep them in sync.

If your application is some sort of master/worker service without real "state", you can do something like this:

1. Send an work item to client

2. Client will send ackowledgement

3. Client will send processing status, say every 30 seconds)

4. When client finished processing work item, it will send result back to server

5. If client goes offline while it has pending work item, server can wait for some time and reschedule it to different client

So, instead of confirming every message and working on protocol level, you're moving to application level and simple heartbeats.

Serge.