Re: [Eventmachine-talk] Checking performance of your EventMachine server??

[Steve Hull]

I think this maybe didn't go through the first time I sent it...

On Wed, May 27, 2009 at 3:06 PM, Steve H <p.w...@gmail.com> wrote:

Hey folks,

I'm working on an EventMachine server as a backend for a realtime game
& chat.  But before we commit to using this in production, we need to
be able to see how it performs.  Is there a recommended way to get
insight into performance of an EventMachine-based program?  For
example, I would like to see bytes sent/received (per socket per
second), open sockets (per second), memory usage, processor usage,
time per request (request = data_received call).

Am I going to have to custom-build this stuff into my server, so I can
get these metrics while load testing?  Do I just need to use the
standard Benchmark library and go with that?  Or is there some either
(1) built-in metrics/monitoring ability within EventMachine, or (2)
gem I could install and leverage?

Thanks,
Steve

[Aman Gupta]

Nothing like this exists yet, but I would love to see a standard set
of benchmarking tools and would definitely consider bundling them with
EM.

Aman

> _______________________________________________
> Eventmachine-talk mailing list
> Eventmac...@rubyforge.org
> http://rubyforge.org/mailman/listinfo/eventmachine-talk
>
_______________________________________________
Eventmachine-talk mailing list
Eventmac...@rubyforge.org
http://rubyforge.org/mailman/listinfo/eventmachine-talk

[Roger Pack]

the only thing I've ever use for profiling is "ab" to test trivial
connection open and closes...tough to tell.
These would indeed be a welcome addition to EM [so you could compare
speed across platforms]
-=r

> insight into performance of an EventMachine-based program?

[Steve Hull]

The one thing I did that I think might be applicable across various implementations is a basic "RPS" (requests per second) counter.  It's a very simple implementation -- in the server module I added an @@rps class variable (which is then incremented in every receive_data call).  Then in my EM::run loop I add a periodic timer that occurs every second which logs the @@rps and zeros it out.  It's not perfect -- the periodic timer doesn't really happen every second, but more like *close* to every second.  But it's good enough for me.

On my new Macbook Pro I have a single server and 100 clients running.  The clients are paired up, sockets kept open, then they send messages back and forth, performing integrity checks (was I the intended recipient?  Was this reply in response to a message I actually sent?  Is the original data intact?).

I ran this for over an hour on my machine.  It averaged about 3000 RPS.  The server process and clients process each took just under 7mb RAM.  The clients process took about 80% CPU, while the server took more like 15% CPU.

All in all pretty dang good.  :)

I think my test has convinced my coworkers and boss -- eventmachine IS up to the task.  We will be building our gameserver on it.

[Aman Gupta]

On Mon, Jun 1, 2009 at 12:29 AM, Steve Hull <p.w...@gmail.com> wrote:
> The one thing I did that I think might be applicable across various
> implementations is a basic "RPS" (requests per second) counter.  It's a very
> simple implementation -- in the server module I added an @@rps class
> variable (which is then incremented in every receive_data call).  Then in my
> EM::run loop I add a periodic timer that occurs every second which logs the
> @@rps and zeros it out.  It's not perfect -- the periodic timer doesn't
> really happen every second, but more like *close* to every second.  But it's
> good enough for me.
> On my new Macbook Pro I have a single server and 100 clients running.  The
> clients are paired up, sockets kept open, then they send messages back and
> forth, performing integrity checks (was I the intended recipient?  Was this
> reply in response to a message I actually sent?  Is the original data
> intact?).

It might be interesting to profile this test to see what was slowing
it down (I've been hacking on a profiler at
http://github.com/tmm1/perftools.rb). Can you share your test?

> I ran this for over an hour on my machine.  It averaged about 3000 RPS.  The
> server process and clients process each took just under 7mb RAM.  The
> clients process took about 80% CPU, while the server took more like 15% CPU.
> All in all pretty dang good.  :)

You can use EM.kqueue=true on OSX and EM.epoll=true on Linux for added
performance gains.

> I think my test has convinced my coworkers and boss -- eventmachine IS up to
> the task.  We will be building our gameserver on it.

Awesome, let us know if you have questions or run into problems.

Aman

[Steve Hull]

> It might be interesting to profile this test to see what was slowing
> it down (I've been hacking on a profiler at
> http://github.com/tmm1/perftools.rb). Can you share your test?

My point in my previous message was that processing 3000 requests per second seems pretty dang good to me.  :)

We decided that if we can get it to handle 200 RPS we should be good for the short-term and mid-term.  Admittedly, my test was super rudimentary, and this is going to slow down a lot when we (inevitably) add in ActiveRecord and some calls to other services, but just as a proof that EventMachine will not be a bottleneck, I think this worked great.

I looked at your perftools.rb project on github.  AWESOME!  I'm not exactly sure how to read the gif outputs, but at first glance, this looks very useful.

[Roger Pack]

> It might be interesting to profile this test to see what was slowing
> it down (I've been hacking on a profiler at
> http://github.com/tmm1/perftools.rb). Can you share your test?

Fascinating. I wonder how the google perftools profiler compares with
gprof straight.
-=r