Best practice for SSL termination with Socket.io

[Matt]

What are people generally using as best practice for SSL termination when they need to use socket.io these days?

I'm usually a big fan of nginx, but hear it just doesn't work with socket.io, and don't want to spend hours getting it there. We have multiple backends that get serviced, and would like some kind of load balancing where possible.

We have enough IPs (I think) that we can host each backend on an individual IP, so we don't need to do name based SSL.

Options I'm considering:

stud -> node

haproxy -> stud -> node

apache -> node (I know everyone hates Apache these days, but it is rock solid, I'm just not sure if it supports what socket.io needs?)

stunnel -> node

Alternatives?

What is best practice?

I don't want to let node do SSL. It's too slow at it (and yes I know this is being worked on), so node-http-proxy is out.

Matt.

[Mikeal Rogers]

We did a full episode of nodeup about this.

http://www.nodeup.com/twentyfour

-Mikeal

--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

[Tim Caswell]

We're using Amazon's ELB in SSL-TCP mode. We then forward this to a
cluster of stateless node http proxies (using node-http-proxy) that do
the host and path based routing to the right app servers. This works
great except we lose the remote client's IP address.

[Guillermo Rauch]

Until ELB adds WebSocket support.

--
Guillermo Rauch

LearnBoost CTO
http://devthought.com

[Jimb Esser]

We're using HAProxy -> stud -> node, and have been pretty happy with it.  Have to jump through a bunch of hoops to get client IP addresses, however.  There was a thread about this a couple weeks ago:

https://groups.google.com/d/topic/nodejs/k0jf4RIVT7E/discussion

  Jimb Esser

  Cloud Party, Inc

[Tim Caswell]

On Fri, Aug 10, 2012 at 3:35 PM, Guillermo Rauch <rau...@gmail.com> wrote:
> Until ELB adds WebSocket support.

The TCP level ELB does, not the HTTP one. But that's why we lose the
client's IP because TCP proxy can't set http headers.

>
> --
> Guillermo Rauch
> LearnBoost CTO
> http://devthought.com
>

[Guillermo Rauch]

That's what I was clarifying. In reality it doesn't really support the WebSocket protocol, and that's why it can't add the X-Forwarded-For. 

It supports it in the same way it supports _any other_ protocol in TCP mode (like XMPP), just forward the traffic along.

[Matt]

So is the only solution that's going to keep the connecting IP going to be using Apache for load balancing and SSL termination? It sounds like there's still no good solution to this.

Matt.

[dvbportal]

> So is the only solution that's going to keep the connecting IP going to be using Apache for load balancing and SSL termination?

No, another solution is HAProxy for balancing and stud for termination. This combination is by far better, if you expect thousands of connections.

[Matt]

On Sun, Aug 12, 2012 at 2:07 PM, dvbportal <dvbp...@gmail.com> wrote:

> So is the only solution that's going to keep the connecting IP going to be using Apache for load balancing and SSL termination?

No, another solution is HAProxy for balancing and stud for termination. This combination is by far better, if you expect thousands of connections.

But loses the connecting IP, right?

[Guillermo Rauch]

No. Stud can use the PROXY protocol to communicate with HAProxy. From the README:

stud will optionally write the client IP address as the first few octets (depending on IPv4 or IPv6) to the backend--or provide that information using HAProxy's PROXY protocol.

[Matt]

Oooh. Done!

[Mikeal Rogers]

Avoid apache, use haproxy or nginx. This configuration, stud and haproxy is quite nice.

I wrote a pure TCP proxy in node rather than using HAProxy.

https://github.com/mikeal/stud-proxy

It works great but I ditch x-forwarded-for and do any IP filtering in stud-proxy rather than the processes behind it. Also, need to make sure that web sockets is wss instead of ws because the backend process think they aren't TLS.

-Mikeal

[Jimb Esser]

Though stud can write the IP info, as far as I can tell there's still no way to pull that out on node without building your own version of node that supports a pre-parse event before passing the stream to the HTTP parser.  This is a pretty simple change and is what we do, but does require our own deployment of node.

Also of note, until 3 days ago, to get stud to *read* the PROXY line with the IP address coming from HAProxy also required a version of stud other than their master one, but it looks like they finally merged one of the outstanding pull requests to add that, so you can use the --proxy-proxy option to pass the IP from HAProxy through stud.

  Jimb Esser

  Cloud Party, Inc

On Monday, August 13, 2012 12:11:17 PM UTC-7, Matt Sergeant wrote:

Oooh. Done!

[Matt]

On Mon, Aug 13, 2012 at 5:43 PM, Jimb Esser <wast...@gmail.com> wrote:

Though stud can write the IP info, as far as I can tell there's still no way to pull that out on node without building your own version of node that supports a pre-parse event before passing the stream to the HTTP parser.  This is a pretty simple change and is what we do, but does require our own deployment of node.

Good to know. I probably don't need it on the node side though. It's mostly just for logs, which stud can write. Am I missing a need for the real IP in node?

 

Also of note, until 3 days ago, to get stud to *read* the PROXY line with the IP address coming from HAProxy also required a version of stud other than their master one, but it looks like they finally merged one of the outstanding pull requests to add that, so you can use the --proxy-proxy option to pass the IP from HAProxy through stud.

Also good to know!

[Jimb Esser]

On Monday, August 13, 2012 3:01:38 PM UTC-7, Matt Sergeant wrote:

Good to know. I probably don't need it on the node side though. It's mostly just for logs, which stud can write. Am I missing a need for the real IP in node?

If the stud logs are enough, and you don't use the IP for application logic, then there's probably no need for it in node.  We need things like being able to ban abusive users by IP.  Also, 90% of the interesting actions a user does on our service happens over the WebSocket link, so we want it for logging as well.

[Matt]

Yes for us it'll only be a minimal percentage, so not a huge deal.

If stud can get the IP via the first few bytes, why can't it then just add an X-Forwarded-For header when proxying to node?

Matt.

[Jimb Esser]

Stud just does SSL termination, absolutely no HTTP parsing, it has no idea if the stream underneath is HTTP or any number of other protocols that get used over SSL, so it has no idea how to add a header into an HTTP stream.  Other more complicated SSL terminators and load balancers do, but they seem to mostly choke on WebSockets which is outside of the usual format of HTTP.

I guess that brings up a good point... stud probably won't do any useful logging for you - it could log IP addresses and connection times, but not individual GET/POSTs, etc, as it has no idea what those are.

[Matt]

Ah the lightbulb is going on now. Thanks.

Maybe Apache would be best (people forget it has a very scalable event based MPM available now).

--

[Matt]

OK, so I finally settled on the following architecture:

 Stud -> HAProxy -> [NGinx ->] Node

Stud does SSL termination.

HAProxy receives the IP address in the first PROXY line from Stud, and performs load balancing with other servers. It will work in both HTTP and TCP modes, HTTP for regular requests on one IP address, and TCP for Socket.io requests on a different IP address (or port, whatever works).

Nginx is in front of the non-socket.io connections only, for those requests HAProxy has added X-Forwarded-For headers. Nginx is there for caching static file requests and transparent gzipping of content.

Thanks a lot for all the help on this thread!

Matt.

[dvbportal]

Great choice. BTW. Have you found a way to dynamically bypass nginx when you detect an upgrade to a websocket connection?

[Matt]

We don't go through nginx for that server - it's bound to a different IP.

So we have basically a dual system setup:

IP1 has the main app server: Stud -> HAProxy in http mode -> Nginx -> Node

IP2 has the socket.io server: Stud -> HAProxy in tcp mode -> Node

The main app server has socket.io loaded so that it can deliver the socket.io.js javascript file to browsers, but the clients actually make the socket.io requests to the server on IP2.

This setup requires haproxy 1.5 (still in dev) for the PROXY protocol support.

We lose the IP address in the socket.io calls, but that's OK - we get to log it for the normal web requests, which are most of the data transfer for our app.

Matt.

On Thu, Aug 16, 2012 at 1:15 PM, dvbportal <dvbp...@gmail.com> wrote:

Great choice. BTW. Have you found a way to dynamically bypass nginx when you detect an upgrade to a websocket connection?

[dvbportal]

> The main app server has socket.io loaded so that it can deliver the socket.io.js javascript file to browsers, but the clients actually make the socket.io requests to the server on IP2.

That is a great idea. Thanks for the details.