Stunnel for high-loaded Faye with SSL (on Node.js)

[Alex Kazeko]

Hi guys,

I want to share my experience about running Faye with SSL. The problem
is that Node and Thin are very slow under HTTPS. So, the idea is to
run Faye with plain connection (http) and setup Stunnel in front of
it.
I've wrote a post about it in my blog:
http://rails-alex.blogspot.com/2011/10/ssl-support-for-high-loaded-faye-nodejs.html.
It also contains examples of Stunnel config.

Alex

[Matthijs Langenberg]

Thank you Alex. 

We are also using Stunnel in production to provide SSL support for Faye. As far as I know Stunnel is one of the few solutions that also supports secure WebSocket.

At the time that I created this setup, Node's SSL support was pretty bad. So I couldn't use that. That is the reason I started using Stunnel, but eventually I want to move away from it.

Stunnel does not use asynchronous IO. By itself that is not a bad thing, but the consequence is that for every connected user, Stunnels forks a child process. This really piles up with many browsers opening a websocket or long-poll connection to Faye.

I actually hoped that the current Node version (v0.4) included good enough SSL support to move away from Stunnel in production. Can you tell me something about this? What were your findings?

Bests,

Matthijs

[Alex Kazeko]

Hi Matthijs,

We use Node 0.4.12 and ssl is still very slow. When I've first deployed to production with SSL enabled (without Stunnel, just SSL on Node) - we had to rollback. 

We had 8 instances of Node running for about 1 000 users online. In 5 minutes we found that it takes few minutes to connect to Faye. Also, New Relic reported that average push to HTTPS Faye takes ~1 sec (we had average response time 80ms and it became 1s).

Now we have up to 6-7k users online (we do up to 25k pushes to Faye from Rails per minute) with next configuration:

- 32 instances of Node with Faye (yep, that's a lot. But it's just for the future)

- 2 Stunnels - 1st is for Nodes 0-15, 2nd is for 16-31. Also, lib-wrap is disabled - it helps a lot.

- ulimit 32000

About 30-40% of users use https.

Load Average on this server is 0.1 - 0.2

Last restart of Stunnel was on October 24 (so, it's running for 9 days) and used 92 minutes of CPU. Same for Node - it used 100-114 minutes of CPU time in 9 days.

We have a cluster of servers so if traffic will grow 10 times - we can increase number of Node servers + we can start moving some of shards to other web servers (now all 32 nodes are running on 1 server we use only for Node, Stunnel, Memcached, Resque and Redis).

BTW, I'm going to create a gem to easily manage Faye shards soon. Will post about it in this group

[Anthony Webb]

Alex,

Curious why you are running so many instances of faye on the same server?  I was under the impression node was so hot and non-blocking that this wasnt necessary in order to scale?  Are there other reasons you are running 32 instances of faye on the same box?

Thanks,

Anthony

[Alex Kazeko]

Anthony,

When we had 8 instances with same traffic - pushes from RoR were slower. Rails do ~25 000 posts to it per minute. It's easier to do it to different Node servers than to one (because of queues). Also, LA was twice higher when we had 8 instances.

I'm not saying we have to run 32 instances, but we have enough memory to do it, so we've chosen that option. Our app is growing pretty good, so we have enough Faye resources. 

Alex

[Anthony Webb]

Thanks for the explanation Alex.  I've been looking at ideas on how to scale out faye.  Right now we are using redis on the backend and have several physical servers each running a single node/faye app.  Hadnt really crossed my mind that spinning up multiple instances of faye on the same server would improve performance like you are seeing.

Can you tell me, are you running a single Node app with that is running several instances of the faye server?  Or is each faye server its own separate node application?  If they are all separate node applications how do you handle spinning them up and making sure they stay online?

[Alex Kazeko]

Anthony,

We are running separate Faye on separate Node. Again, because of request queues. 

We don't use Redis - it loads server a little bit more. We removed Redis because in our application users are connected to 2 channels: "/#{user.id}" and "/global". In 99.9% requests we send data to user's channel by posting data to needed shard. In 0.01% of cases we have to push data to all 32 servers, so we do it via Resque. I shortly wrote about it in my blog: http://rails-alex.blogspot.com/2011/10/high-performance-publishsubscribe.html. If in your application most communication is done via global channels - my schema won't work. 

I'm going to write a gem to manage Faye shards soon. Will post about it as soon as ready.

Alex

[James Coglan]

On 1 November 2011 08:23, Alex Kazeko <kazekoa...@gmail.com> wrote:

http://rails-alex.blogspot.com/2011/10/high-performance-publishsubscribe.html

"Before describing Faye I want to say that Juggernaut has 1 great benefit which Faye doesn't have - you can push events directly to Redis and Juggernaut will catch it and process. You can't easily do the same with Faye (maybe it will be done in future). Instead of it you have to send a HTTP request, which is slower and loads Faye's server."

This is technically possible, although how to do it is probably not at all obvious. I think I implemented it on a branch a while ago but now I can't find it. It was weird and I probably got rid of it. Anyway, consider the local transport, used to send messages between a client and server in the same process:

module Faye

  class Transport::Local < Transport

    def self.usable?(endpoint)

      endpoint.is_a?(Server)

    end

    

    def batching?

      false

    end

    

    def request(message, timeout)

      @endpoint.process(message, true) { |responses| receive(responses) }

    end

  end

  

  Transport.register 'in-process', Transport::Local

end

What we want is a client that can talk directly to Redis. Transports accept client messages and deal with sending them to the server, and giving the response back to the client, as you can see above. To make a Redis transport, what we do is we construct a Server object using the Redis engine *within* our transport, and send messages to that. No HTTP traffic is involved; the Server translates messages into Redis commands for you and it just works.

module Faye

  class Transport::Redis < Transport

    def self.usable?(endpoint)

      endpoint.is_a?(Hash) and endpoint[:host] and endpoint[:port]

    end

    

    def batching?

      false

    end

    

    def request(message, timeout)

      server.process(message, true) { |responses| receive(responses) }

    end

    

    def server

      @server ||= Faye::Server.new(

                    :engine => {

                      :type => 'redis',

                      :host => @endpoint[:host],

                      :port => @endpoint[:port]

                    }

                  )

    end

  end

  

  Transport.register 'redis', Transport::Redis

end

You also need to make some changes to Server to support this, in order to adjust how transport negotiation works, but that's the basic idea. I ended up taking this idea out of the codebase because it was too hard for the user to make this transport's internal server act the same as a normal server, in terms of running the same extensions, etc. But if you want to have a crack at it be my guest.

[Alex Kazeko]

Yep, that's why I wrote "You can't easily do the same with Faye (maybe it will be done in future)" ;)

I've been reading server code to find out if there is an easy way to do a direct push to Redis, but I didn't have enough time to finish it because of other activities. Hopefully I'll get back to it in future. 

[Matthijs Langenberg]

I am only running one Node instance. 

There is one Stunnel instance, which has 400 child processes. So I guess there are about 400 online users.

I would like to try Node's current SSL support sometime. Maybe the traffic that I am having isn't causing any issues.

That way I could have two application servers, with both 1 Node instance handle about 200 users.