SockJS-Tornado not receiving msgs behind HAProxy

Showing 1-16 of 16 messages
SockJS-Tornado not receiving msgs behind HAProxy Chris 10/21/12 1:15 PM

I am deploying my application, and am able to establish connection with a sockjs-tornado server behind haproxy. According to the front-end js log, message is sent after connection but the server-side never received anything and no error is produced. In addition, shortly after sending a message, disconnection happened (not sure why)

I put logging on the first line inside on_message method on the server-side, nothing showed up in the tornado log. I also checked the haproxy log, this is what i saw:

Oct 21 15:49:27 localhost.localdomain haproxy[6494]: 129.xx.xxx.xxx:54794 [21/Oct/2012:15:49:26.823] public www/www1 580/0/0/0/580 200 373 - - ---- 5/5/0/1/0 0/0 "GET /sockjs/wait/info HTTP/1.1"
Oct 21 15:49:42 localhost.localdomain haproxy[6494]: 129.xx.xxx.xxx:54804 [21/Oct/2012:15:49:27.563] public www/www1 0/0/0/0/15002 101 132 - - cD-- 0/0/0/0/0 0/0 "GET /sockjs/wait/746/s49uf3iy/websocket HTTP/1.1"

I guess the first one is related to establishing connection, and the second one is related to the message I sent from the front-end.

Any guess what is going on here? And is there a sockjs-tornado specific log I can check?
Re: [tornado] SockJS-Tornado not receiving msgs behind HAProxy Serge S. Koval 10/21/12 1:59 PM
Are you using this haproxy configuration file: https://github.com/sockjs/sockjs-node/blob/master/examples/haproxy.cfg?

However:
1. If your Tornado is running on 10.0.0.2:8080 and you're matching /sockjs/wait/ as SockJS route, make sure you can access http://10.0.0.2:8080/sockjs/wait/
2. Overall, it appears that websocket connection was established - it was able to get SockJS information (first call) and establish websocket connection (second call). You can set logging level to debug to see sockjs-tornado logs: 

    import logging
    logging.getLogger().setLevel(logging.DEBUG)

Hope it helps.

Serge.
Re: [tornado] SockJS-Tornado not receiving msgs behind HAProxy Chris 10/21/12 2:59 PM

Hello Serge, 

My config is basically based on the one you quoted, except that I don't have option http-pretend-keepalive as it causes problem with nginx, which is also behind the haproxy. This is what I have (the relevant parts):

defaults
        log             global
        mode            http
        option          httplog
        option          dontlognull
        retries         3
        option          redispatch
        option          http-server-close
        maxconn         500
        contimeout      5s
        clitimeout      10s
        srvtimeout      10s

frontend public
        mode    http
        bind    *:80
        timeout client  120s
        option  forwardfor

backend www
        timeout server 600s
        server content 127.0.0.1:8080

I don't think i'm running Tornado at 10.0.0.2:8080 (Is what you see in the log above?), but at 127.0.0.1:8080.  And I can access http://mysite.com/sockjs/wait/, it returns "Welcome to SockJS". I will tweak the logging and see what else I can find. But if you see anything suspicious so far, do give me a shout!

On Sunday, October 21, 2012 4:59:06 PM UTC-4, Serge S. Koval wrote:
Are you using this haproxy configuration file: https://github.com/sockjs/sockjs-node/blob/master/examples/haproxy.cfg?

However:
1. If your Tornado is running on 10.0.0.2:8080 and you're matching /sockjs/wait/ as SockJS route, make sure you can access http://10.0.0.2:8080/sockjs/wait/
2. Overall, it appears that websocket connection was established - it was able to get SockJS information (first call) and establish websocket connection (second call). You can set logging level to debug to see sockjs-tornado logs: 

    import logging
    logging.getLogger().setLevel(logging.DEBUG)

Hope it helps.

Serge.
Re: [tornado] SockJS-Tornado not receiving msgs behind HAProxy Serge S. Koval 10/21/12 3:11 PM
Hi,

 My comments are inline.

On Mon, Oct 22, 2012 at 12:59 AM, Chris <bhp...@gmail.com> wrote:
My config is basically based on the one you quoted, except that I don't have option http-pretend-keepalive as it causes problem with nginx, which is also behind the haproxy. This is what I have (the relevant parts):
1. I had problem with older haproxy version, when it did not honor "balance uri depth 2" if http-pretent-keepalive was disabled and was using round robin balancing. Make sure that all subsequent requests for polling transports go to same Tornado backend;
2. I don't see ACLs for SockJS in your configuration. Like here: https://github.com/sockjs/sockjs-node/blob/master/examples/haproxy.cfg#L18

I don't think i'm running Tornado at 10.0.0.2:8080 (Is what you see in the log above?), but at 127.0.0.1:8080.  
No, that IP was used to illustrate what to check. 127.0.0.1 is fine as well.

And I can access http://mysite.com/sockjs/wait/, it returns "Welcome to SockJS". I will tweak the logging and see what else I can find. But if you see anything suspicious so far, do give me a shout!
OK, it means that haproxy routes HTTP requests to your SockJS backend. Try enabling logging, maybe connection getting closed due to some kind of error, etc.
 
Serge.
Re: [tornado] SockJS-Tornado not receiving msgs behind HAProxy Chris 10/21/12 3:29 PM

I'm using haproxy 1.4.22 (latest stable), and i guess i don't need round-robin balancing here, as there is only a single sockjs backend.

I don't have a specific ACL for this sockjs-tornado server, as it also has some other simple RequestHandlers. Basically, I just say:

default_backend www

and all requests (both https requests and sockjs msgs) go to this sockjs backend. Is that right?

One more details (if it matters), in the front-end js code, I'm trying to send this kind of msg: 

conn.onopen = function() {
console.log("connected");
// send WAIT_MSG
var wait_msg = JSON.stringify({"id": 101, "content": "blahblahblah"});
conn.send(wait_msg);
console.log("WAIT_MSG sent");
};

I've tried enabling the debugging log like this:

if __name__ == "__main__":
    options.parse_command_line()
    logging.getLogger().setLevel(logging.DEBUG)
    http_server = tornado.httpserver.HTTPServer(Application())
    http_server.listen(options.port)
    tornado.ioloop.IOLoop.instance().start()

But still don't see anything except for the initial connection information in the log. After the above js code sent the msg, nothing is received by on_message(self, msg) method in sockjs-tornado, and no other error appears.
Re: [tornado] SockJS-Tornado not receiving msgs behind HAProxy Chris 10/22/12 4:39 AM

I have noticed another thing, the second sockjs-tornado msg in haproxy log (which suggests that websocket connection is established):

Oct 21 15:49:42 localhost.localdomain haproxy[6494]: 129.xx.xxx.xxx:54804 [21/Oct/2012:15:49:27.563] public www/www1 0/0/0/0/15002 101 132 - - cD-- 0/0/0/0/0 0/0 "GET /sockjs/wait/746/s49uf3iy/websocket HTTP/1.1"

only appears in the log when I shutdown the sockjs-tornado backend process. Before I do that, there is only one sockjs-tornado related msg in haproxy log:

Oct 21 15:49:27 localhost.localdomain haproxy[6494]: 129.xx.xxx.xxx:54794 [21/Oct/2012:15:49:26.823] public www/www1 580/0/0/0/580 200 373 - - ---- 5/5/0/1/0 0/0 "GET /sockjs/wait/info HTTP/1.1"

and there is no other error msg or disconnection, the sockjs-tornado just waits and waits even though a msg is sent from the front-end. This is very strange.
Re: [tornado] SockJS-Tornado not receiving msgs behind HAProxy Serge S. Koval 10/22/12 6:23 AM
Hi,

On Mon, Oct 22, 2012 at 1:29 AM, Chris <bhp...@gmail.com> wrote:
and all requests (both https requests and sockjs msgs) go to this sockjs backend. Is that right?
Ah, sure. Yes, it will redirect all requests to single Tornado instance.

If you use Chrome, open network tab in development tools, find running websocket connection and check if something being received from client-side or server-side. You should see incoming 'o' and your outgoing message with some SockJS protocol framing.

Unfortunately, I ran out of ideas for now. If it works without haproxy (you can connect with SockJS to 127.0.0.1:8080 and does not work with haproxy), it has something to do with haproxy.

Also, just saw it - increase client timeout to something like 120s, because SockJS has long-polling protocols and you don't want haproxy to close long running connections after 5s.

Serge. 
Re: [tornado] SockJS-Tornado not receiving msgs behind HAProxy Chris 10/23/12 5:46 AM

Hi Serge,

Yes, I can confirm that it works on localhost but not behind haproxy. I also used the Chrome network tool to inspect the websocket connection, this is what I saw:

  1. Request URL:
  2. Request Method:
    GET
  3. Status Code:
    101 Switching Protocols
  4. Request Headersview source
    1. Connection:
      Upgrade
    2. Host:
    3. Origin:
    4. Sec-WebSocket-Extensions:
      x-webkit-deflate-frame
    5. Sec-WebSocket-Key:
      E7uYzLwfLi/SawcGBcGEig==
    6. Sec-WebSocket-Version:
      13
    7. Upgrade:
      websocket
    8. (Key3):
      00:00:00:00:00:00:00:00
  5. Response Headersview source
    1. Connection:
      Upgrade
    2. Sec-WebSocket-Accept:
      JzEG6Q5bwLIf4oVykcSkNZhEkpA=
    3. Upgrade:
      websocket
    4. (Challenge Response):
      00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00

And both the 'Type' and 'Time Latency' for this websocket in the Chrome Network tool just says 'Pending', which may explain why the sockjs-tornado backend never receives the msg, because I suspect the websocket handshake is not completed (even though according to SockJS client, apparently conn.readyState === SockJS.OPEN, at this point). This may also be related to why I don't see:

Oct 23 08:40:17 localhost.localdomain haproxy[14121]: 129.67.137.105:54632 [23/Oct/2012:08:38:55.192] public www/content 0/0/0/0/82147 101 141 - - CD-- 4/4/0/0/0 0/0 "GET /sockjs/wait/307/gxrzu0e6/websocket HTTP/1.1" 

in the haproxy log until I shutdown the sockjs-tornado backend. So maybe something is wrong between haproxy and the sockjs-tornado backend, which prevents the handshake to succeed?
Re: [tornado] SockJS-Tornado not receiving msgs behind HAProxy Serge S. Koval 10/23/12 6:53 AM
OK, more questions:

1. What's your OS?
2. If you use Windows, any antivirus software?
3. Can you move your haproxy port to something else than port 80? Try moving to port 5000 and see if it starts working.
4. If it is not windows, any corporate firewalls/proxies?

It looks like there's something in between you and the server, which does not support websockets. I saw this behavior with poorly coded antiviruses (Avast, etc) or corporate transparent proxies.

Serge.
Re: [tornado] SockJS-Tornado not receiving msgs behind HAProxy Chris 10/23/12 7:18 AM

1. What's your OS?

Mac OS X 10.6.8

4. If it is not windows, any corporate firewalls/proxies?

I have this 'Symantec Endpoint Protection' installed, but I don't think it is an issue, as when I go on to: http://websocketstest.com/, everything is ticked. 

3. Can you move your haproxy port to something else than port 80? Try moving to port 5000 and see if it starts working.
 
Yes, that works! I changed to access everything through port 5000, and now websocket works! But why?!

I start suspecting whether it is due to some firewall settings on my server that is hosting haproxy and the sockjs-tornado backend. It is running Ubuntu 10.04 on Linode, and its iptables config are:

*filter

#  Allows all loopback (lo0) traffic and drop all traffic to 127/8 that doesn't use lo0
-A INPUT -i lo -j ACCEPT
-A INPUT -i lo -d 127.0.0.0/8 -j REJECT

#  Accepts all established inbound connections
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

#  Allows all outbound traffic
#  You can modify this to only allow certain traffic
-A OUTPUT -j ACCEPT

#  Allows HTTP and HTTPS connections from anywhere (the normal ports for websites)
-A INPUT -p tcp --dport 80 -j ACCEPT
-A INPUT -p tcp --dport 5000 -j ACCEPT
-A INPUT -p tcp --dport 443 -j ACCEPT

#  Allows SSH connections
#
#  THE -dport NUMBER IS THE SAME ONE YOU SET UP IN THE SSHD_CONFIG FILE
#
-A INPUT -p tcp -m state --state NEW --dport 22 -j ACCEPT

#  Allow ping
-A INPUT -p icmp -m icmp --icmp-type 8 -j ACCEPT

#  log iptables denied calls
-A INPUT -m limit --limit 5/min -j LOG --log-prefix "iptables denied: " --log-level 7

#  Reject all other inbound - default deny unless explicitly allowed policy
-A INPUT -j REJECT
-A FORWARD -j REJECT

COMMIT
Re: [tornado] SockJS-Tornado not receiving msgs behind HAProxy Serge S. Koval 10/23/12 7:37 AM
Hi,

I have this 'Symantec Endpoint Protection' installed, but I don't think it is an issue, as when I go on to: http://websocketstest.com/, everything is ticked. 
Even for port 80? Maybe this website is in exclusion list for your SEP, no idea.

Try:
a) Disabling it (if you can) and try again;
b) Access from location that does not have Symantec Endpoint Protection.
 
Yes, that works! I changed to access everything through port 5000, and now websocket works! But why?!
Something between your browser and haproxy messes up websocket connection. I have suspicion that this is Symantec Endpoint Protection. 
 
I start suspecting whether it is due to some firewall settings on my server that is hosting haproxy and the sockjs-tornado backend. It is running Ubuntu 10.04 on Linode, and its iptables config are:
It looks good to me. So I doubt it has something to do with your server.

Serge. 
Re: [tornado] SockJS-Tornado not receiving msgs behind HAProxy Chris 10/23/12 9:18 AM

Hello Serge,

Something between your browser and haproxy messes up websocket connection. I have suspicion that this is Symantec Endpoint Protection.

I've just tried from another network (with Symantec Endpoint Protection still on my laptop), and it works fine with haproxy through port 80. So it must be something that is specific to the network I've been on which prevents websocket handshake through port 80, and that was a college LAN.

Given that we can't control user's network environment, does it mean that we should always communicate with sockjs-tornado backends using non-80 port? If that is the case, how can we expose sockjs-tornado backends on a different port (say 5000) but on the same domain? For example, sockjs-tornado backends can be reached in browsers at www.mysite.com:5000/sockjs, while everything else is accessible using www.mysite.com/blah/blah?

Thank you.
Re: [tornado] SockJS-Tornado not receiving msgs behind HAProxy Serge S. Koval 10/23/12 9:30 AM
Ah, then it is caused by something in the college network then.

I escalated this to SockJS developer as well - maybe he'll have some ideas how to handle it in the future. Client is not handling edge case when it received something from the server (that's why readyState == OPEN), but can't send anything to the server.

Best way to avoid it right now:
1. Host it on different port
2. Use SSL (same as #1 and additional encryption)

Yes, you can expose sockjs-tornado on different port - SockJS will work in cross-domain scenario (different port is treated as different domain by browser) on all supported browsers. See here: https://github.com/sockjs/sockjs-client#supported-transports-by-browser-html-served-from-http-or-https

Serge.
Re: [tornado] SockJS-Tornado not receiving msgs behind HAProxy Chris 10/23/12 10:19 AM

Hello Serge,

Thanks again for all the useful info. About your suggestions below:

Best way to avoid it right now:
1. Host it on different port
2. Use SSL (same as #1 and additional encryption)

I want to confirm that whether you meant I should host haproxy on a different port than 80? If so, doesn't it mean that users need to use www.mysite.com:port_no/blah in the browser to access the site? And by SSL, do you mean run a dev version of haproxy (1.5+) with SSL enabled? So instead of using port 80, now websocket traffic go through port 443, and because it is now encrypted, it should have a less chance of being interrupted?

Thank you


 
Re: [tornado] SockJS-Tornado not receiving msgs behind HAProxy Serge S. Koval 10/23/12 10:54 AM
Hi,

On Tue, Oct 23, 2012 at 8:19 PM, Chris <bhp...@gmail.com> wrote:
I want to confirm that whether you meant I should host haproxy on a different port than 80?
You can make haproxy listen on two ports: on port 80 for HTTP traffic, different port for SockJS-related traffic. 

If so, doesn't it mean that users need to use www.mysite.com:port_no/blah in the browser to access the site?
Only your application will know that it should connect to different port for real-time portion of the website. For rest of the world it will be ordinary website working on default port.
 
And by SSL, do you mean run a dev version of haproxy (1.5+) with SSL enabled?
Yes, terminating SSL on haproxy end. Usually firewalls don't mess with SSL traffic, unless it is big corporate firewall which might hijack SSL session.
 
So instead of using port 80, now websocket traffic go through port 443, and because it is now encrypted, it should have a less chance of being interrupted?
That's correct.

Serge.
Re: [tornado] SockJS-Tornado not receiving msgs behind HAProxy Chris 10/23/12 11:48 AM

Thanks very much, Serge!