How scalable is the EM chat server example?

[Henry]

Thank you (Aman?) for this example: http://gist.github.com/89588 of a
very easy to understand chat server.

I want to create a game server based on this type of broadcast
communication between players. Each game room has only 4 players, and
there will probably be only 1 message broadcast amongst the players
per second, but I want the application to be able to handle thousands
of separate and simultaneous game room instances.

In the EM chat room example, the connections are more or less held in
memory and kept open for the duration of the connection. Is there any
way to scale this application similarly to the way mongrel or
passenger "workers" get behind a load balancer of some kind? Can the
connection object be marshalled and stored into a shared database so
that separate servers can divide and conquer the requests? Or is
there a way to route connection requests to the same server if they
are of the same game? Just thinking aloud here, sorry if I'm way off
the mark or don't make sense. I don't know too much about proxying or
networking.

Thanks in advance.

[James Tucker]

On 16 Oct 2010, at 02:51, Henry wrote:

> I want to create a game server based on this type of broadcast
> communication between players. Each game room has only 4 players, and
> there will probably be only 1 message broadcast amongst the players
> per second, but I want the application to be able to handle thousands
> of separate and simultaneous game room instances.

That will work fine. We support tens of thousands of connections at the EM level[1]. What you will find is that walking the broadcast lists, generating strings, doing authentication and routing logic, etc on the ruby side will start to be the source of slow down eventually. Many people doing long running web requests type work aim for around 4k connections per ruby process, however, depending on the exact pattern it's certainly possible to push this number up higher.

You probably don't want to push that limit too high purely because the reactor loop is single threaded. If it takes 1ms for your app logic to generate a message for a single game and send to each of the 4 players, then 4000 connections will take up all of your available CPU time, and with overhead and stutter, you'll start to see a little bit of lag at that point (assuming the game is as you say, generating one message per second, and there are 4 players per game).

This all being said, 4000 concurrently active players is quite a lot, for a new service. I hope you do find that much success, but you may want to build for practicality first, and then approach the scaling side of these issues later, if/when it is appropriate.

> In the EM chat room example, the connections are more or less held in
> memory and kept open for the duration of the connection. Is there any
> way to scale this application similarly to the way mongrel or
> passenger "workers" get behind a load balancer of some kind?

Yes, you could use a proxy and a routing key. If you're dividing up by room, you could have a message header that includes the room id, and route to app server instances using that. You might want to look at something like proxymachine when you get to this.

> Can the
> connection object be marshalled and stored into a shared database so
> that separate servers can divide and conquer the requests?

No. The problem with that is, you need to be able to share connection fd's, which is not really supported by OSes in this way, although it's possible to do something plan9-y and attach sockets to fifos or the like, then push and pull data from "files". You'd end up with a bit of a painful scheduler and a bunch of extra work to get this going on most common OSes, and it's not worth the trouble. I'm also not entirely persuaded it'll scale all that well.

> Or is
> there a way to route connection requests to the same server if they
> are of the same game? Just thinking aloud here, sorry if I'm way off
> the mark or don't make sense. I don't know too much about proxying or
> networking.

As stated, take a look at proxymachine, all you'll need is a header to use as a routing key and you should be good to go, at least for your early spike tests. I doubt you'll need this for a very long time though.

[1] http://github.com/raggi/em-scale-test

[Homan Chou]

Wow. Thank you very much for the in depth answer! It's very
informative. Maybe EM (level1?) will be enough for my purposes then,
but I will probably use an html5 enabled browser as the client and for
the server I'll use eventmachine/em-socket to create a websocket
server. There won't be *too* much logic that happens on the ruby
side, mainly just when one player makes a move, the other players are
broadcast the update so they are all in-sync. I suppose one easy way
to scale is simply have multiple websocket servers, each one having up
to 4k connections. When a player logs in on a home page, they can be
redirected to a another page which uses the websocket server with the
least load. Is this similar the proxy method you mentioned? I'll
take a look at proxymachine.

Regarding your mention that the reactor core is single threaded...
doesn't using EM.defer make it make it multi-threaded, or perhaps only
if used with jruby?

> --
> You received this message because you are subscribed to the Google Groups "EventMachine" group.
> To post to this group, send email to eventm...@googlegroups.com.
> To unsubscribe from this group, send email to eventmachine...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/eventmachine?hl=en.
>
>

[Mike Perham]

A friend recently open sourced Holla:

http://github.com/maccman/holla

which can be modified to do what you want.

[James Tucker]

On 17 Oct 2010, at 03:27, Homan Chou wrote:

> Regarding your mention that the reactor core is single threaded...
> doesn't using EM.defer make it make it multi-threaded, or perhaps only
> if used with jruby?

defer is purely a ruby side convenience api, it does not affect the reactor directly. the threads used there are used exclusively for running defer procs.

[Ian Ragsdale]

Connections themselves can't be shared amongst different servers, but it's pretty easy to proxy a connection from one server to another in eventmachine. There is a single method that can pass all data that comes in on one connection to another, which makes it fairly trivial to write a simple proxy server.

That said, while all our network servers are written in EM, we use HAProxy to distribute connections across boxes. I've been using it in production for a while now, and it's been 100% reliable and trouble-free. I set it up once and it just runs. In the last 24 hours our main instance has processed over 660,000 incoming connections using 27MB of ram and almost no CPU. You could absolutely write something in EM to do the same thing, but you'd have to maintain it yourself, and it would be tough to beat the performance.

- Ian