Faye behind an HTTP proxy

[John Pignata]

I'm working on deploying a Faye-based server and have had significant
trouble getting it to work behind a load balancer with either web
sockets or long polling. The symptoms are lost messages and lagged
connections (that is - messages are delayed up to the client timeout
period.) I'm seeing very similar behavior behind Amazon's Elastic Load
Balancer (TCP/TLS) and HAProxy. Are there any users who have deployed
a Faye service behind a load balancer and if so are there any gotchas?

[James Coglan]

I'm not an expert on HAProxy by any means but I do know that it *can* work. Here are a couple of configs I use for testing:

https://github.com/faye/faye/blob/master/examples/shared/haproxy.conf

https://github.com/faye/faye-websocket-node/blob/master/examples/haproxy.conf

Whether you're using tcp or http mode makes a difference, and indeed faye-websocket handles some problems with WebSockets when running behind HAProxy. faye-0.8 will begin using this code when it's released in the new year.

Still, even if sockets don't work, polling should be fine. Run HAProxy's stats module, check the backends are responding, and check that TCP/HTTP is not being killed off prematurely before the Faye server can return a response. You can try reducing the timeout setting on your Faye server to see if this makes a difference.

[John Pignata]

Thanks - very helpful. Lowering the timeout from 300 -> 60 seems to
significantly improve the situation. Ideally we'd like a really long
server-side timeout to reduce connection overhead so we'll need to
continue tinkering. Is there anything inherently unsafe about setting
a super-long timeout on the server side?

Thanks again,
-jp

On Dec 22, 3:55 pm, James Coglan <jcog...@gmail.com> wrote:

> On 22 December 2011 20:50, John Pignata <j...@pignata.com> wrote:
>
> > I'm working on deploying a Faye-based server and have had significant
> > trouble getting it to work behind a load balancer with either web
> > sockets or long polling. The symptoms are lost messages and lagged
> > connections (that is - messages are delayed up to the client timeout
> > period.) I'm seeing very similar behavior behind Amazon's Elastic Load
> > Balancer (TCP/TLS) and HAProxy. Are there any users who have deployed
> > a Faye service behind a load balancer and if so are there any gotchas?
>
> I'm not an expert on HAProxy by any means but I do know that it *can* work.
> Here are a couple of configs I use for testing:
>

> https://github.com/faye/faye/blob/master/examples/shared/haproxy.confhttps://github.com/faye/faye-websocket-node/blob/master/examples/hapr...

[James Coglan]

On 22 December 2011 22:29, John Pignata <jo...@pignata.com> wrote:

Is there anything inherently unsafe about setting
a super-long timeout on the server side?

The only risks are that intermediaries may close the connection before Faye sends a response -- this of course depends on your load-balancer settings and anything else you put between the Faye server and client. On the client side, all transport types except JSON-P can detect network errors -- JSON-P has to rely on timeouts so tends to be less responsive in dealing with errors. These days, it tends to only be Opera and old (< 4.0) versions of Firefox that still use JSON-P, and they only use it for cross-domain connections.

[John Pignata]

I've gotten it to work pretty well using stud (https://github.com/
bumptech/stud) and HAProxy using your sample config. Thanks again for
that. Amazon ELB turned out to be a no-go. I'm not sure why as it
should be doing the same amount of work that stud is doing.

I have one remaining issue that I'm digging into where Safari seems to
lose connection permanently sometime during the first couple of
minutes. It seems to connect successfully via WebSockets but within a
few minutes I get this CORS-related error in the console:

XMLHttpRequest cannot load https://push.domain.com/faye. Origin
https://domain.com is not allowed by Access-Control-Allow-Origin.

.. and the client never attempts reconnect. I don't see anything
especially interesting in the debug logs. OPTIONS for the /faye
endpoint is returning a Access-Control-Allow-Origin header with * as
expected. Is this a known issue with Safari/WebSockets/CORS flakiness?

-jp

On Dec 22, 5:36 pm, James Coglan <jcog...@gmail.com> wrote:

[John Pignata]

Alternatively I sometimes see:

GET
https://push.domain.com/faye?message=%5B%7B%22channel%22%3A%22%2Fmeta%2Fconnect%22%2C%22clientId%22%3A%221v1jvu80c4ai5k1fcrz9805uh9v6%22%2C%22connectionType%22%3A%22callback-polling%22%2C%22id%22%3A%224%22%7D%5D&jsonp=__jsonp2__
502 (Bad Gateway)

But it's always at the two minute mark of being idle. Not sure why it
would drop to jsonp. Any pointers would be appreciated.

Thanks again,
-jp

On Dec 23, 11:18 am, John Pignata <j...@pignata.com> wrote:
> I've gotten it to work pretty well using stud (https://github.com/
> bumptech/stud) and HAProxy using your sample config. Thanks again for
> that. Amazon ELB turned out to be a no-go. I'm not sure why as it
> should be doing the same amount of work that stud is doing.
>
> I have one remaining issue that I'm digging into where Safari seems to
> lose connection permanently sometime during the first couple of
> minutes. It seems to connect successfully via WebSockets but within a
> few minutes I get this CORS-related error in the console:
>

> XMLHttpRequest cannot loadhttps://push.domain.com/faye. Originhttps://domain.comis not allowed by Access-Control-Allow-Origin.

[John Pignata]

I unfortunately still observed reliability issues behind HAProxy aside
from the aforementioned Safari pain. I've rewritten the service to
bind to HTTP/HTTPS directly and have kicked all proxies to the curb.
It's much happier this way.

jcoglan - Thanks for the help and for your work on Faye!

-jp

On Dec 23, 11:33 am, John Pignata <j...@pignata.com> wrote:
> Alternatively I sometimes see:
>
> GEThttps://push.domain.com/faye?message=%5B%7B%22channel%22%3A%22%2Fmeta...

> 502 (Bad Gateway)
>
> But it's always at the two minute mark of being idle. Not sure why it
> would drop to jsonp. Any pointers would be appreciated.
>
> Thanks again,
> -jp
>
> On Dec 23, 11:18 am, John Pignata <j...@pignata.com> wrote:
>
>
>
>
>
>
>
> > I've gotten it to work pretty well using stud (https://github.com/
> > bumptech/stud) and HAProxy using your sample config. Thanks again for
> > that. Amazon ELB turned out to be a no-go. I'm not sure why as it
> > should be doing the same amount of work that stud is doing.
>
> > I have one remaining issue that I'm digging into where Safari seems to
> > lose connection permanently sometime during the first couple of
> > minutes. It seems to connect successfully via WebSockets but within a
> > few minutes I get this CORS-related error in the console:
>

> > XMLHttpRequest cannot loadhttps://push.domain.com/faye. Originhttps://domain.comisnot allowed by Access-Control-Allow-Origin.

[James Coglan]

On 23 December 2011 16:18, John Pignata <jo...@pignata.com> wrote:

I have one remaining issue that I'm digging into where Safari seems to
lose connection permanently sometime during the first couple of
minutes. It seems to connect successfully via WebSockets but within a
few minutes I get this CORS-related error in the console:

XMLHttpRequest cannot load https://push.domain.com/faye. Origin
https://domain.com is not allowed by Access-Control-Allow-Origin.

.. and the client never attempts reconnect. I don't see anything
especially interesting in the debug logs. OPTIONS for the /faye
endpoint is returning a Access-Control-Allow-Origin header with * as
expected. Is this a known issue with Safari/WebSockets/CORS flakiness?

I've not seen this before. I'm wondering whether more Access-Control stuff is needed on the response. Faye returns these fields for OPTIONS:

      'Access-Control-Allow-Origin':      '*',

      'Access-Control-Allow-Credentials': 'false',

      'Access-Control-Max-Age':           '86400',

      'Access-Control-Allow-Methods':     'POST, GET, PUT, DELETE, OPTIONS',

      'Access-Control-Allow-Headers':     'Accept, Content-Type, X-Requested-With'

And it returns this on *all* requests if Origin is sent:

      headers['Access-Control-Allow-Origin'] = origin 

I wondering if it needs to echo more than just the origin in some cases to work (some user agents use OPTIONS to find out if CORS is allowed, some use headers on the resource itself).

When you say the client never attempts to reconnect, do you mean that it just never tries this particular request again, or can you not send messages to that client at all? I'm wondering if what you're seeing is the last /meta/connect request made with the default transport before it upgrades to WebSockets.

[James Coglan]

On 23 December 2011 16:33, John Pignata <jo...@pignata.com> wrote:

Alternatively I sometimes see:

GET
https://push.domain.com/faye?message=%5B%7B%22channel%22%3A%22%2Fmeta%2Fconnect%22%2C%22clientId%22%3A%221v1jvu80c4ai5k1fcrz9805uh9v6%22%2C%22connectionType%22%3A%22callback-polling%22%2C%22id%22%3A%224%22%7D%5D&jsonp=__jsonp2__
502 (Bad Gateway)

But it's always at the two minute mark of being idle. Not sure why it
would drop to jsonp. Any pointers would be appreciated.

This is a problem in your server-side setup. Make sure no proxies or load balancers are terminating the connection before Faye gets a chance to respond. 

[James Coglan]

On 24 December 2011 05:24, John Pignata <jo...@pignata.com> wrote:

I unfortunately still observed reliability issues behind HAProxy aside
from the aforementioned Safari pain. I've rewritten the service to
bind to HTTP/HTTPS directly and have kicked all proxies to the curb.
It's much happier this way.

Ah, fair enough :) 

jcoglan - Thanks for the help and for your work on Faye!

Not at all, thanks for getting in touch.