Hello world performance

[Roman Bataev]

Hi.

I wanted to reproduce a test described here http://dosync.posterous.com/22397098
On my MacBook Pro node.js managed to handle ~6.9K req/s, while aleph
is only able to handle 1.3K req/s.
I am using the latest snapshot of aleph from github, and HTTP server
hello world snippet from here https://github.com/ztellman/aleph

Am I doing something wrong or is it expected performance for the
latest snapshot?

[Zach Tellman]

I haven't tested the performance in a while, but that's a pretty
serious regression. Are you giving the JVM time to warm up?

Zach

[David Nolen]

I just ran the hello-world bench on my machine using the latest version of Aleph w/ enough runs for the JVM to be warm and I'm seeing around 2900+ req/sec. That's quite a bit slower than what I used to see.

A quick run under VisualVM shows that a good chunk of time is eaten up in reflection.

[Zach Tellman]

I've pushed a version that should be markedly faster, and I think
there's still some room for improvement. Let me know if you're not
seeing any change.

Zach

[David Nolen]

ab -n 10000 -c 5 -k http://localhost:8080/

Gives just above 5000 req/s now. Nice!

VisualVM stills shows 15% or so of the time being spent in reflection. Eliminating that should get us competitive again w/ Node.js for The World's Most Pointless Benchmark ;)

David

[Roman Bataev]

Yesterday's version (up to "forgot an argument" commit) gave ~3.4K req/
seq on my machine. Big performance boost compared to previous version,
but node.js is still twice as fast.
With the last commit ("clean up optimizations") it gives ~1.9K req/
seq.

Also, every other time I run

ab -n 5000 -c 5 http://127.0.0.1:8080/

I get terrible result like this:

Percentage of the requests served within a certain time (ms)
50% 2
66% 2
75% 3
80% 3
90% 4
95% 8
98% 11
99% 12
100% 6979 (longest request)

So basically it freezes for several seconds on some requests. Any
ideas what could be causing this?

On Nov 9, 11:37 am, David Nolen <dnolen.li...@gmail.com> wrote:
> ab -n 10000 -c 5 -khttp://localhost:8080/

>
> Gives just above 5000 req/s now. Nice!
>
> VisualVM stills shows 15% or so of the time being spent in reflection.
> Eliminating that should get us competitive again w/ Node.js for The World's
> Most Pointless Benchmark ;)
>
> David
>
>
>
>
>
>
>
> On Tue, Nov 9, 2010 at 3:11 AM, Zach Tellman <ztell...@gmail.com> wrote:
> > I've pushed a version that should be markedly faster, and I think
> > there's still some room for improvement.  Let me know if you're not
> > seeing any change.
>
> > Zach
>

> > On Mon, Nov 8, 2010 at 7:52 PM, David Nolen <dnolen.li...@gmail.com>

> > wrote:
> > > I just ran the hello-world bench on my machine using the latest version
> > of
> > > Aleph w/ enough runs for the JVM to be warm and I'm seeing around 2900+
> > > req/sec. That's quite a bit slower than what I used to see.
> > > A quick run under VisualVM shows that a good chunk of time is eaten up in
> > > reflection.

> > > On Mon, Nov 8, 2010 at 10:38 PM, Zach Tellman <ztell...@gmail.com>

> > wrote:
>
> > >> I haven't tested the performance in a while, but that's a pretty
> > >> serious regression.  Are you giving the JVM time to warm up?
>
> > >> Zach
>

> > >> On Mon, Nov 8, 2010 at 6:08 PM, Roman Bataev <roman.bat...@gmail.com>

> > >> wrote:
> > >> > Hi.
>
> > >> > I wanted to reproduce a test described here
> > >> >http://dosync.posterous.com/22397098
> > >> > On my MacBook Pro node.js managed to handle ~6.9K req/s, while aleph
> > >> > is only able to handle 1.3K req/s.
> > >> > I am using the latest snapshot of aleph from github, and HTTP server

> > >> > hello world snippet from  herehttps://github.com/ztellman/aleph

[Alex Miller]

Sounds like GC to me.  Did you run with:

-verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails

These settings are designed to be run in production settings so shouldn't significantly impact performance.  

I suspect you'll see a full gc cycle at some point (probably about oh 6-7 seconds long). 

---

From: Roman Bataev <roman....@gmail.com>
To: Aleph <alep...@googlegroups.com>
Sent: Wed, November 10, 2010 8:48:26 PM
Subject: Re: Hello world performance

[Roman Bataev]

Please disregard my complain about freezing. Apparently I have the
same issue with node.js, so it's not a JVM or aleph issue. Sorry about
that.

On Nov 10, 10:13 pm, Alex Miller <alexdmil...@yahoo.com> wrote:
> Sounds like GC to me.  Did you run with:
>
> -verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails
>
> These settings are designed to be run in production settings so shouldn't
> significantly impact performance.  
>
> I suspect you'll see a full gc cycle at some point (probably about oh 6-7
> seconds long).  
>
> ________________________________

> From: Roman Bataev <roman.bat...@gmail.com>

> To: Aleph <alep...@googlegroups.com>
> Sent: Wed, November 10, 2010 8:48:26 PM
> Subject: Re: Hello world performance
>
> Yesterday's version (up to "forgot an argument" commit) gave ~3.4K req/
> seq on my machine. Big performance boost compared to previous version,
> but node.js is still twice as fast.
> With the last commit ("clean up optimizations") it gives ~1.9K req/
> seq.
>
> Also, every other time I run
>

> ab -n 5000 -c 5http://127.0.0.1:8080/

[David Nolen]

On Wed, Nov 10, 2010 at 9:48 PM, Roman Bataev <roman....@gmail.com> wrote:

Yesterday's version (up to "forgot an argument" commit) gave ~3.4K req/
seq on my machine. Big performance boost compared to previous version,
but node.js is still twice as fast.
With the last commit ("clean up optimizations") it gives ~1.9K req/
seq.

Also, every other time I run

ab -n 5000 -c 5 http://127.0.0.1:8080/

I get terrible result like this:

Percentage of the requests served within a certain time (ms)
 50%      2
 66%      2
 75%      3
 80%      3
 90%      4
 95%      8
 98%     11
 99%     12
 100%   6979 (longest request)

So basically it freezes for several seconds on some requests. Any
ideas what could be causing this?

Ephemeral port setting for your OS needs to be tweaked. node.js has this problem as well.

I agree that it would be good to remove all reflection in a library like Aleph. I don't think it's very important that we precisely match or beat node.js for the hello world benchmark as on more realistic benchmarks node.js tends suffer compared to Clojure. For one Clojure's function call performance (the building block of any application) simply puts v8 to shame.

David

[Zach Tellman]

Can you confirm that you have the latest version of Lamina? Running lein clean and deleting ~/.m2 should suffice. There shouldn't have been a sizable regression between those commits.

[Roman Bataev]

I have the latest snapshot of Lamina
lamina-0.3.1-20101010.033133-3.jar. I just double checked my tests:
current trunk: ~1.9K req/s
after reverting last commit ("clean up optimizations"): ~3.5K req/s

The same version of Lamina was used in both cases.

On Nov 10, 10:22 pm, Zach Tellman <ztell...@gmail.com> wrote:
> Can you confirm that you have the latest version of Lamina?  Running lein clean and deleting ~/.m2 should suffice. There shouldn't have been a sizable regression between those commits.
>

> On Nov 10, 2010, at 6:48 PM, Roman Bataev <roman.bat...@gmail.com> wrote:
>
>
>
>
>
>
>
> > Yesterday's version (up to "forgot an argument" commit) gave ~3.4K req/
> > seq on my machine. Big performance boost compared to previous version,
> > but node.js is still twice as fast.
> > With the last commit ("clean up optimizations") it gives ~1.9K req/
> > seq.
>
> > Also, every other time I run
>

> > ab -n 5000 -c 5http://127.0.0.1:8080/

[Roman Bataev]

Thank you for the tip about ephemeral port setting. I got this fixed
now.

I think it's pretty important to match or be at least very close to
node.js on the hello world benchmark. I understand that it is not
realistic, but it is very easy to reproduce and it gives you a general
impression about the performance of a platform, a base line so to
speak. I am a newcomer to both platforms and I am still trying to
decide which one I would like to invest more time in. Basic hello
world benchmark can be one of the factor that will affect my decision.
So if aleph community wants to grow matching (or beating) competitors
in basic benchmark may help.

On Nov 10, 10:19 pm, David Nolen <dnolen.li...@gmail.com> wrote:

> On Wed, Nov 10, 2010 at 9:48 PM, Roman Bataev <roman.bat...@gmail.com>wrote:
>
>
>
>
>
>
>
>
>
> > Yesterday's version (up to "forgot an argument" commit) gave ~3.4K req/
> > seq on my machine. Big performance boost compared to previous version,
> > but node.js is still twice as fast.
> > With the last commit ("clean up optimizations") it gives ~1.9K req/
> > seq.
>
> > Also, every other time I run
>

> > ab -n 5000 -c 5http://127.0.0.1:8080/

[David Nolen]

On Fri, Nov 12, 2010 at 10:52 PM, Roman Bataev <roman....@gmail.com> wrote:

Thank you for the tip about ephemeral port setting. I got this fixed
now.

I think it's pretty important to match or be at least very close to
node.js on the hello world benchmark. I understand that it is not
realistic, but it is very easy to reproduce and it gives you a general
impression about the performance of a platform, a base line so to
speak. I am a newcomer to both platforms and I am still trying to
decide which one I would like to invest more time in. Basic hello
world benchmark can be one of the factor that will affect my decision.
So if aleph community wants to grow matching (or beating) competitors
in basic benchmark may help.

I don't see this regression in performance. I'm using the latest snapshots of aleph 0.1.3 and lamina 0.3.1.

I think it's good to hear this kind of feedback from people interested in Aleph. I agree that Aleph should at least be competitive w/ Node.js on this baseline. 

Zach, do you have any thoughts on this?

David

[Zach Tellman]

I was able to reproduce the regression between the speed-up and
clean-up on a Core2Duo system, but not on an i5. I don't know exactly
why this is, but the important thing is that there was a regression,
and it's fixed now.

With respect to the general question of performance, I'm of two minds.
I agree speed is important, and ignoring performance issues that are
trivial to fix isn't constructive. I'm glad that the regression in
performance was reported, and I should probably have some automated
benchmarks so that I'm not just testing correctness before committing
changes.

On the other hand, I don't know how meaningful the 'Hello World'
benchmark is when evaluating a library for real-world use. As the
amount of work per request increases, the communication overhead
measured by the benchmark becomes increasingly insignificant. I'd put
more weight on how simple it is to write the evented application
logic, as long as there aren't serious performance concerns.

I'll also note that Clojure code has a lot of variables, few of which
are well understood. I'm told, but have not tested, that
-XX:MaxInlineSize=100 can greatly improve performance. I suspect, but
have not confirmed, that 1.3 will improve performance quite a bit.
There are a hundred other JVM knobs to tweak, none of which I have any
expertise in. Once this becomes more than just a theoretical issue, I
think that the community will adopt some best practices.

Hopefully that answers everyone's questions. Let me know if it doesn't.

Zach