comparison of web servers REDUX and pull request?

[Bip Thelin]

Hi,

 a few weeks ago we did a test of how the different web servers on Chicago Boss behaved. Since then we have done some changes and now have a tree with changes that perform a bit better. On an average we see a ~40% better throughput. Better performance is good, right? well, even if the changes aren't substantial they change a bit of the overall design which might not be desirable. So before I push a pull request I'd like to get a first heads up if this is a direction that is suitable/good for CB.

You can read the updated test at: http://labs.evolope.com/post/16866763467/a-comparison-of-web-servers-for-running-chicago-boss

and you can view the changes to CB here: https://github.com/bipthelin/ChicagoBoss/tree/optz

There's of course a lot more that can be done but with this last test we see that we can get pretty dependable performance without any errors being thrown and that is really important (at least for us).

Overall I want to say that CB is really well laid out and have a good architecture. So it was hard to find any crazy gains due to bad code/design.

--
Bip Thelin
 
Evolope AB | Lugnets Allé 1 | 120 33 Stockholm
Tel 08-533 335 37 | Mob 0735-18 18 90
www.evolope.se

[ll...@writersglen.com]

Hi Bip,

As a noobie to CB with minimal technical qualifications, this kind of solid engineering evaluation is profoundly reassuring. Performance, reliability, and supportability are all high on my list as I set out to build the applications to support my publishing business.

I do have a question about changing gen-servers to modules, however: As I understand it, the system-at-large is protected from failure of a well-supervised gen-server. Would this still be true if you changed existing gen-servers to modules?

Thanks for your fine work.

LRP

[Evan Miller]

If sessions are still the bottleneck you might try adding more session
processes to deal with concurrent connections. Right now there is just
one registered gen_server process which handles all session requests
(and which blocks while awaiting a reply from the session storage
layer).

I think the underlying problem might not be the number of calls to
gen_server per se but the fact that CB uses single registered
processes in many places so that the message mailboxes get backed up.
A more OTP solution would be to dispatch requests to worker processes
that returned values to the requester when ready. BossNews uses this
architecture, where each channel is managed by a worker process. A
central boss_news registered process simply dispatches messages to the
appropriate channel, and the channel manager sends the reply to the
requester when the work is done.

Did you try the SMP test I suggested, i.e. run the tests again with
SMP disabled (-smp disable)? This should reveal whether the problem is
really CPU time spent in gen_server or if it is due to poor process
design. I suspect the latter.

Nice find with max_connections. BTW I'd prefer it if max_connections
were made a configuration option, ideally that applied to all servers
(even though we might dump non-Misultin servers in the future).

So to summarize: I like the parameter tuning, but I'm not convinced
we've found the real bottleneck with gen_server. I think when we do,
we'll see the system scale as well as standalone Misultin.

Thanks for your all your work on this!

Evan

--
Evan Miller
http://www.evanmiller.org/

[Bip Thelin]

Valid question Lloyd. Without getting in to the details, the changes I proposed should not in any way make CB less fault tolerant. I'm however underway in trying to find alternative ways to achieve higher performance.

--
Bip Thelin
 
Evolope AB | Lugnets Allé 1 | 120 33 Stockholm
Tel 08-533 335 37 | Mob 0735-18 18 90
www.evolope.se

[ll...@writersglen.com]

You're my man.

Thanks, Bip.

[Bip Thelin]

Hi Evan,

 Yes I did try the SMP test and as you guessed we're definitely process bound and the problem is definitely in that the gen_server mailboxes get clogged up. Especially since there's a lot of serialized synchronous gen_server calls going on. I tried that another approach that provided some performance improvements and that was to use gen_server deferring such as:

handle_call({xxx, yyy}, From, State) ->
  spawn_link(fun() ->
      Result = slow_function(State),
      gen_server:reply(From, Result)
    end),
  {noreply, State};

This worked reasonably well for longer running handle_call's. Unfortunately there aren't that many/any of them and if ended up eating any non blocking performance gains with the added overhead.

Could you please elaborate a bit more about your idea so I can take a stab at testing it.

--
Bip Thelin
 
Evolope AB | Lugnets Allé 1 | 120 33 Stockholm
Tel 08-533 335 37 | Mob 0735-18 18 90
www.evolope.se

[Evan Miller]

Hi Bip,

The Poolboy library might help:

https://github.com/devinus/poolboy

The basic idea is to use worker processes with one central process
that doles out work to them (e.g. retrieving session info from
Memcached). This way the central process always returns immediately
and can process the next message in its mailbox, while the workers hum
away on the blocking activity.

I haven't used it, but it looks like Poolboy will take care of the
details for you. It may not help with mock sessions (yet) but should
help with Memcached sessions. If we want to improve the scalability of
mock sessions we might use a hashing algorithm, so that there are many
stateful session processes each of which hold a subset of the
sessions. Then a worker pool will make sense.

By the way I think BossDB would be a great candidate for using
Poolboy, and it would let us rip out the internal pools used by the DB
adapters. Reworking BossDB to use a worker pool is one of the last
objectives for 1.0.

Let me know if there's anything I can clarify further, and thanks for

all your work on this!

Evan