I wanted to reproduce a test described here http://dosync.posterous.com/22397098 On my MacBook Pro node.js managed to handle ~6.9K req/s, while aleph
is only able to handle 1.3K req/s.
I am using the latest snapshot of aleph from github, and HTTP server
hello world snippet from here https://github.com/ztellman/aleph
Am I doing something wrong or is it expected performance for the
latest snapshot?
On Mon, Nov 8, 2010 at 6:08 PM, Roman Bataev <roman.bat...@gmail.com> wrote: > Hi.
> I wanted to reproduce a test described here http://dosync.posterous.com/22397098 > On my MacBook Pro node.js managed to handle ~6.9K req/s, while aleph > is only able to handle 1.3K req/s. > I am using the latest snapshot of aleph from github, and HTTP server > hello world snippet from here https://github.com/ztellman/aleph
> Am I doing something wrong or is it expected performance for the > latest snapshot?
I just ran the hello-world bench on my machine using the latest version of Aleph w/ enough runs for the JVM to be warm and I'm seeing around 2900+ req/sec. That's quite a bit slower than what I used to see.
A quick run under VisualVM shows that a good chunk of time is eaten up in reflection.
On Mon, Nov 8, 2010 at 10:38 PM, Zach Tellman <ztell...@gmail.com> wrote: > I haven't tested the performance in a while, but that's a pretty > serious regression. Are you giving the JVM time to warm up?
> Zach
> On Mon, Nov 8, 2010 at 6:08 PM, Roman Bataev <roman.bat...@gmail.com> > wrote: > > Hi.
> > I wanted to reproduce a test described here > http://dosync.posterous.com/22397098 > > On my MacBook Pro node.js managed to handle ~6.9K req/s, while aleph > > is only able to handle 1.3K req/s. > > I am using the latest snapshot of aleph from github, and HTTP server > > hello world snippet from here https://github.com/ztellman/aleph
> > Am I doing something wrong or is it expected performance for the > > latest snapshot?
I've pushed a version that should be markedly faster, and I think there's still some room for improvement. Let me know if you're not seeing any change.
On Mon, Nov 8, 2010 at 7:52 PM, David Nolen <dnolen.li...@gmail.com> wrote: > I just ran the hello-world bench on my machine using the latest version of > Aleph w/ enough runs for the JVM to be warm and I'm seeing around 2900+ > req/sec. That's quite a bit slower than what I used to see. > A quick run under VisualVM shows that a good chunk of time is eaten up in > reflection. > On Mon, Nov 8, 2010 at 10:38 PM, Zach Tellman <ztell...@gmail.com> wrote:
>> I haven't tested the performance in a while, but that's a pretty >> serious regression. Are you giving the JVM time to warm up?
>> Zach
>> On Mon, Nov 8, 2010 at 6:08 PM, Roman Bataev <roman.bat...@gmail.com> >> wrote: >> > Hi.
>> > I wanted to reproduce a test described here >> > http://dosync.posterous.com/22397098 >> > On my MacBook Pro node.js managed to handle ~6.9K req/s, while aleph >> > is only able to handle 1.3K req/s. >> > I am using the latest snapshot of aleph from github, and HTTP server >> > hello world snippet from here https://github.com/ztellman/aleph
>> > Am I doing something wrong or is it expected performance for the >> > latest snapshot?
VisualVM stills shows 15% or so of the time being spent in reflection. Eliminating that should get us competitive again w/ Node.js for The World's Most Pointless Benchmark ;)
On Tue, Nov 9, 2010 at 3:11 AM, Zach Tellman <ztell...@gmail.com> wrote: > I've pushed a version that should be markedly faster, and I think > there's still some room for improvement. Let me know if you're not > seeing any change.
> Zach
> On Mon, Nov 8, 2010 at 7:52 PM, David Nolen <dnolen.li...@gmail.com> > wrote: > > I just ran the hello-world bench on my machine using the latest version > of > > Aleph w/ enough runs for the JVM to be warm and I'm seeing around 2900+ > > req/sec. That's quite a bit slower than what I used to see. > > A quick run under VisualVM shows that a good chunk of time is eaten up in > > reflection. > > On Mon, Nov 8, 2010 at 10:38 PM, Zach Tellman <ztell...@gmail.com> > wrote:
> >> I haven't tested the performance in a while, but that's a pretty > >> serious regression. Are you giving the JVM time to warm up?
> >> Zach
> >> On Mon, Nov 8, 2010 at 6:08 PM, Roman Bataev <roman.bat...@gmail.com> > >> wrote: > >> > Hi.
> >> > I wanted to reproduce a test described here > >> > http://dosync.posterous.com/22397098 > >> > On my MacBook Pro node.js managed to handle ~6.9K req/s, while aleph > >> > is only able to handle 1.3K req/s. > >> > I am using the latest snapshot of aleph from github, and HTTP server > >> > hello world snippet from here https://github.com/ztellman/aleph
> >> > Am I doing something wrong or is it expected performance for the > >> > latest snapshot?
Yesterday's version (up to "forgot an argument" commit) gave ~3.4K req/
seq on my machine. Big performance boost compared to previous version,
but node.js is still twice as fast.
With the last commit ("clean up optimizations") it gives ~1.9K req/
seq.
> VisualVM stills shows 15% or so of the time being spent in reflection.
> Eliminating that should get us competitive again w/ Node.js for The World's
> Most Pointless Benchmark ;)
> David
> On Tue, Nov 9, 2010 at 3:11 AM, Zach Tellman <ztell...@gmail.com> wrote:
> > I've pushed a version that should be markedly faster, and I think
> > there's still some room for improvement. Let me know if you're not
> > seeing any change.
> > Zach
> > On Mon, Nov 8, 2010 at 7:52 PM, David Nolen <dnolen.li...@gmail.com>
> > wrote:
> > > I just ran the hello-world bench on my machine using the latest version
> > of
> > > Aleph w/ enough runs for the JVM to be warm and I'm seeing around 2900+
> > > req/sec. That's quite a bit slower than what I used to see.
> > > A quick run under VisualVM shows that a good chunk of time is eaten up in
> > > reflection.
> > > On Mon, Nov 8, 2010 at 10:38 PM, Zach Tellman <ztell...@gmail.com>
> > wrote:
> > >> I haven't tested the performance in a while, but that's a pretty
> > >> serious regression. Are you giving the JVM time to warm up?
> > >> Zach
> > >> On Mon, Nov 8, 2010 at 6:08 PM, Roman Bataev <roman.bat...@gmail.com>
> > >> wrote:
> > >> > Hi.
> > >> > I wanted to reproduce a test described here
> > >> >http://dosync.posterous.com/22397098 > > >> > On my MacBook Pro node.js managed to handle ~6.9K req/s, while aleph
> > >> > is only able to handle 1.3K req/s.
> > >> > I am using the latest snapshot of aleph from github, and HTTP server
> > >> > hello world snippet from herehttps://github.com/ztellman/aleph
> > >> > Am I doing something wrong or is it expected performance for the
> > >> > latest snapshot?
These settings are designed to be run in production settings so shouldn't significantly impact performance.
I suspect you'll see a full gc cycle at some point (probably about oh 6-7 seconds long).
________________________________ From: Roman Bataev <roman.bat...@gmail.com> To: Aleph <aleph-lib@googlegroups.com> Sent: Wed, November 10, 2010 8:48:26 PM Subject: Re: Hello world performance
Yesterday's version (up to "forgot an argument" commit) gave ~3.4K req/ seq on my machine. Big performance boost compared to previous version, but node.js is still twice as fast. With the last commit ("clean up optimizations") it gives ~1.9K req/ seq.
> VisualVM stills shows 15% or so of the time being spent in reflection. > Eliminating that should get us competitive again w/ Node.js for The World's > Most Pointless Benchmark ;)
> David
> On Tue, Nov 9, 2010 at 3:11 AM, Zach Tellman <ztell...@gmail.com> wrote: > > I've pushed a version that should be markedly faster, and I think > > there's still some room for improvement. Let me know if you're not > > seeing any change.
> > Zach
> > On Mon, Nov 8, 2010 at 7:52 PM, David Nolen <dnolen.li...@gmail.com> > > wrote: > > > I just ran the hello-world bench on my machine using the latest version > > of > > > Aleph w/ enough runs for the JVM to be warm and I'm seeing around 2900+ > > > req/sec. That's quite a bit slower than what I used to see. > > > A quick run under VisualVM shows that a good chunk of time is eaten up in > > > reflection. > > > On Mon, Nov 8, 2010 at 10:38 PM, Zach Tellman <ztell...@gmail.com> > > wrote:
> > >> I haven't tested the performance in a while, but that's a pretty > > >> serious regression. Are you giving the JVM time to warm up?
> > >> Zach
> > >> On Mon, Nov 8, 2010 at 6:08 PM, Roman Bataev <roman.bat...@gmail.com> > > >> wrote: > > >> > Hi.
> > >> > I wanted to reproduce a test described here > > >> >http://dosync.posterous.com/22397098 > > >> > On my MacBook Pro node.js managed to handle ~6.9K req/s, while aleph > > >> > is only able to handle 1.3K req/s. > > >> > I am using the latest snapshot of aleph from github, and HTTP server > > >> > hello world snippet from herehttps://github.com/ztellman/aleph
> > >> > Am I doing something wrong or is it expected performance for the > > >> > latest snapshot?
> These settings are designed to be run in production settings so shouldn't
> significantly impact performance.
> I suspect you'll see a full gc cycle at some point (probably about oh 6-7
> seconds long).
> ________________________________
> From: Roman Bataev <roman.bat...@gmail.com>
> To: Aleph <aleph-lib@googlegroups.com>
> Sent: Wed, November 10, 2010 8:48:26 PM
> Subject: Re: Hello world performance
> Yesterday's version (up to "forgot an argument" commit) gave ~3.4K req/
> seq on my machine. Big performance boost compared to previous version,
> but node.js is still twice as fast.
> With the last commit ("clean up optimizations") it gives ~1.9K req/
> seq.
> > VisualVM stills shows 15% or so of the time being spent in reflection.
> > Eliminating that should get us competitive again w/ Node.js for The World's
> > Most Pointless Benchmark ;)
> > David
> > On Tue, Nov 9, 2010 at 3:11 AM, Zach Tellman <ztell...@gmail.com> wrote:
> > > I've pushed a version that should be markedly faster, and I think
> > > there's still some room for improvement. Let me know if you're not
> > > seeing any change.
> > > Zach
> > > On Mon, Nov 8, 2010 at 7:52 PM, David Nolen <dnolen.li...@gmail.com>
> > > wrote:
> > > > I just ran the hello-world bench on my machine using the latest version
> > > of
> > > > Aleph w/ enough runs for the JVM to be warm and I'm seeing around 2900+
> > > > req/sec. That's quite a bit slower than what I used to see.
> > > > A quick run under VisualVM shows that a good chunk of time is eaten up in
> > > > reflection.
> > > > On Mon, Nov 8, 2010 at 10:38 PM, Zach Tellman <ztell...@gmail.com>
> > > wrote:
> > > >> I haven't tested the performance in a while, but that's a pretty
> > > >> serious regression. Are you giving the JVM time to warm up?
> > > >> Zach
> > > >> On Mon, Nov 8, 2010 at 6:08 PM, Roman Bataev <roman.bat...@gmail.com>
> > > >> wrote:
> > > >> > Hi.
> > > >> > I wanted to reproduce a test described here
> > > >> >http://dosync.posterous.com/22397098 > > > >> > On my MacBook Pro node.js managed to handle ~6.9K req/s, while aleph
> > > >> > is only able to handle 1.3K req/s.
> > > >> > I am using the latest snapshot of aleph from github, and HTTP server
> > > >> > hello world snippet from herehttps://github.com/ztellman/aleph
> > > >> > Am I doing something wrong or is it expected performance for the
> > > >> > latest snapshot?
> Yesterday's version (up to "forgot an argument" commit) gave ~3.4K req/ > seq on my machine. Big performance boost compared to previous version, > but node.js is still twice as fast. > With the last commit ("clean up optimizations") it gives ~1.9K req/ > seq.
> Percentage of the requests served within a certain time (ms) > 50% 2 > 66% 2 > 75% 3 > 80% 3 > 90% 4 > 95% 8 > 98% 11 > 99% 12 > 100% 6979 (longest request)
> So basically it freezes for several seconds on some requests. Any > ideas what could be causing this?
Ephemeral port setting for your OS needs to be tweaked. node.js has this problem as well.
I agree that it would be good to remove all reflection in a library like Aleph. I don't think it's very important that we precisely match or beat node.js for the hello world benchmark as on more realistic benchmarks node.js tends suffer compared to Clojure. For one Clojure's function call performance (the building block of any application) simply puts v8 to shame.
Can you confirm that you have the latest version of Lamina? Running lein clean and deleting ~/.m2 should suffice. There shouldn't have been a sizable regression between those commits.
On Nov 10, 2010, at 6:48 PM, Roman Bataev <roman.bat...@gmail.com> wrote:
> Yesterday's version (up to "forgot an argument" commit) gave ~3.4K req/ > seq on my machine. Big performance boost compared to previous version, > but node.js is still twice as fast. > With the last commit ("clean up optimizations") it gives ~1.9K req/ > seq.
> Percentage of the requests served within a certain time (ms) > 50% 2 > 66% 2 > 75% 3 > 80% 3 > 90% 4 > 95% 8 > 98% 11 > 99% 12 > 100% 6979 (longest request)
> So basically it freezes for several seconds on some requests. Any > ideas what could be causing this?
> On Nov 9, 11:37 am, David Nolen <dnolen.li...@gmail.com> wrote: >> ab -n 10000 -c 5 -khttp://localhost:8080/
>> Gives just above 5000 req/s now. Nice!
>> VisualVM stills shows 15% or so of the time being spent in reflection. >> Eliminating that should get us competitive again w/ Node.js for The World's >> Most Pointless Benchmark ;)
>> David
>> On Tue, Nov 9, 2010 at 3:11 AM, Zach Tellman <ztell...@gmail.com> wrote: >>> I've pushed a version that should be markedly faster, and I think >>> there's still some room for improvement. Let me know if you're not >>> seeing any change.
>>> Zach
>>> On Mon, Nov 8, 2010 at 7:52 PM, David Nolen <dnolen.li...@gmail.com> >>> wrote: >>>> I just ran the hello-world bench on my machine using the latest version >>> of >>>> Aleph w/ enough runs for the JVM to be warm and I'm seeing around 2900+ >>>> req/sec. That's quite a bit slower than what I used to see. >>>> A quick run under VisualVM shows that a good chunk of time is eaten up in >>>> reflection. >>>> On Mon, Nov 8, 2010 at 10:38 PM, Zach Tellman <ztell...@gmail.com> >>> wrote:
>>>>> I haven't tested the performance in a while, but that's a pretty >>>>> serious regression. Are you giving the JVM time to warm up?
>>>>> Zach
>>>>> On Mon, Nov 8, 2010 at 6:08 PM, Roman Bataev <roman.bat...@gmail.com> >>>>> wrote: >>>>>> Hi.
>>>>>> I wanted to reproduce a test described here >>>>>> http://dosync.posterous.com/22397098 >>>>>> On my MacBook Pro node.js managed to handle ~6.9K req/s, while aleph >>>>>> is only able to handle 1.3K req/s. >>>>>> I am using the latest snapshot of aleph from github, and HTTP server >>>>>> hello world snippet from herehttps://github.com/ztellman/aleph
>>>>>> Am I doing something wrong or is it expected performance for the >>>>>> latest snapshot?
I have the latest snapshot of Lamina
lamina-0.3.1-20101010.033133-3.jar. I just double checked my tests:
current trunk: ~1.9K req/s
after reverting last commit ("clean up optimizations"): ~3.5K req/s
The same version of Lamina was used in both cases.
On Nov 10, 10:22 pm, Zach Tellman <ztell...@gmail.com> wrote:
> Can you confirm that you have the latest version of Lamina? Running lein clean and deleting ~/.m2 should suffice. There shouldn't have been a sizable regression between those commits.
> On Nov 10, 2010, at 6:48 PM, Roman Bataev <roman.bat...@gmail.com> wrote:
> > Yesterday's version (up to "forgot an argument" commit) gave ~3.4K req/
> > seq on my machine. Big performance boost compared to previous version,
> > but node.js is still twice as fast.
> > With the last commit ("clean up optimizations") it gives ~1.9K req/
> > seq.
> > Percentage of the requests served within a certain time (ms)
> > 50% 2
> > 66% 2
> > 75% 3
> > 80% 3
> > 90% 4
> > 95% 8
> > 98% 11
> > 99% 12
> > 100% 6979 (longest request)
> > So basically it freezes for several seconds on some requests. Any
> > ideas what could be causing this?
> > On Nov 9, 11:37 am, David Nolen <dnolen.li...@gmail.com> wrote:
> >> ab -n 10000 -c 5 -khttp://localhost:8080/
> >> Gives just above 5000 req/s now. Nice!
> >> VisualVM stills shows 15% or so of the time being spent in reflection.
> >> Eliminating that should get us competitive again w/ Node.js for The World's
> >> Most Pointless Benchmark ;)
> >> David
> >> On Tue, Nov 9, 2010 at 3:11 AM, Zach Tellman <ztell...@gmail.com> wrote:
> >>> I've pushed a version that should be markedly faster, and I think
> >>> there's still some room for improvement. Let me know if you're not
> >>> seeing any change.
> >>> Zach
> >>> On Mon, Nov 8, 2010 at 7:52 PM, David Nolen <dnolen.li...@gmail.com>
> >>> wrote:
> >>>> I just ran the hello-world bench on my machine using the latest version
> >>> of
> >>>> Aleph w/ enough runs for the JVM to be warm and I'm seeing around 2900+
> >>>> req/sec. That's quite a bit slower than what I used to see.
> >>>> A quick run under VisualVM shows that a good chunk of time is eaten up in
> >>>> reflection.
> >>>> On Mon, Nov 8, 2010 at 10:38 PM, Zach Tellman <ztell...@gmail.com>
> >>> wrote:
> >>>>> I haven't tested the performance in a while, but that's a pretty
> >>>>> serious regression. Are you giving the JVM time to warm up?
> >>>>> Zach
> >>>>> On Mon, Nov 8, 2010 at 6:08 PM, Roman Bataev <roman.bat...@gmail.com>
> >>>>> wrote:
> >>>>>> Hi.
> >>>>>> I wanted to reproduce a test described here
> >>>>>>http://dosync.posterous.com/22397098 > >>>>>> On my MacBook Pro node.js managed to handle ~6.9K req/s, while aleph
> >>>>>> is only able to handle 1.3K req/s.
> >>>>>> I am using the latest snapshot of aleph from github, and HTTP server
> >>>>>> hello world snippet from herehttps://github.com/ztellman/aleph
> >>>>>> Am I doing something wrong or is it expected performance for the
> >>>>>> latest snapshot?
Thank you for the tip about ephemeral port setting. I got this fixed
now.
I think it's pretty important to match or be at least very close to
node.js on the hello world benchmark. I understand that it is not
realistic, but it is very easy to reproduce and it gives you a general
impression about the performance of a platform, a base line so to
speak. I am a newcomer to both platforms and I am still trying to
decide which one I would like to invest more time in. Basic hello
world benchmark can be one of the factor that will affect my decision.
So if aleph community wants to grow matching (or beating) competitors
in basic benchmark may help.
On Nov 10, 10:19 pm, David Nolen <dnolen.li...@gmail.com> wrote:
> On Wed, Nov 10, 2010 at 9:48 PM, Roman Bataev <roman.bat...@gmail.com>wrote:
> > Yesterday's version (up to "forgot an argument" commit) gave ~3.4K req/
> > seq on my machine. Big performance boost compared to previous version,
> > but node.js is still twice as fast.
> > With the last commit ("clean up optimizations") it gives ~1.9K req/
> > seq.
> > Percentage of the requests served within a certain time (ms)
> > 50% 2
> > 66% 2
> > 75% 3
> > 80% 3
> > 90% 4
> > 95% 8
> > 98% 11
> > 99% 12
> > 100% 6979 (longest request)
> > So basically it freezes for several seconds on some requests. Any
> > ideas what could be causing this?
> Ephemeral port setting for your OS needs to be tweaked. node.js has this
> problem as well.
> I agree that it would be good to remove all reflection in a library like
> Aleph. I don't think it's very important that we precisely match or beat
> node.js for the hello world benchmark as on more realistic benchmarks
> node.js tends suffer compared to Clojure. For one Clojure's function call
> performance (the building block of any application) simply puts v8 to shame.
On Fri, Nov 12, 2010 at 10:52 PM, Roman Bataev <roman.bat...@gmail.com>wrote:
> Thank you for the tip about ephemeral port setting. I got this fixed > now.
> I think it's pretty important to match or be at least very close to > node.js on the hello world benchmark. I understand that it is not > realistic, but it is very easy to reproduce and it gives you a general > impression about the performance of a platform, a base line so to > speak. I am a newcomer to both platforms and I am still trying to > decide which one I would like to invest more time in. Basic hello > world benchmark can be one of the factor that will affect my decision. > So if aleph community wants to grow matching (or beating) competitors > in basic benchmark may help.
I don't see this regression in performance. I'm using the latest snapshots of aleph 0.1.3 and lamina 0.3.1.
I think it's good to hear this kind of feedback from people interested in Aleph. I agree that Aleph should at least be competitive w/ Node.js on this baseline.
I was able to reproduce the regression between the speed-up and clean-up on a Core2Duo system, but not on an i5. I don't know exactly why this is, but the important thing is that there was a regression, and it's fixed now.
With respect to the general question of performance, I'm of two minds. I agree speed is important, and ignoring performance issues that are trivial to fix isn't constructive. I'm glad that the regression in performance was reported, and I should probably have some automated benchmarks so that I'm not just testing correctness before committing changes.
On the other hand, I don't know how meaningful the 'Hello World' benchmark is when evaluating a library for real-world use. As the amount of work per request increases, the communication overhead measured by the benchmark becomes increasingly insignificant. I'd put more weight on how simple it is to write the evented application logic, as long as there aren't serious performance concerns.
I'll also note that Clojure code has a lot of variables, few of which are well understood. I'm told, but have not tested, that -XX:MaxInlineSize=100 can greatly improve performance. I suspect, but have not confirmed, that 1.3 will improve performance quite a bit. There are a hundred other JVM knobs to tweak, none of which I have any expertise in. Once this becomes more than just a theoretical issue, I think that the community will adopt some best practices.
Hopefully that answers everyone's questions. Let me know if it doesn't.
On Fri, Nov 12, 2010 at 8:16 PM, David Nolen <dnolen.li...@gmail.com> wrote: > On Fri, Nov 12, 2010 at 10:52 PM, Roman Bataev <roman.bat...@gmail.com> > wrote:
>> Thank you for the tip about ephemeral port setting. I got this fixed >> now.
>> I think it's pretty important to match or be at least very close to >> node.js on the hello world benchmark. I understand that it is not >> realistic, but it is very easy to reproduce and it gives you a general >> impression about the performance of a platform, a base line so to >> speak. I am a newcomer to both platforms and I am still trying to >> decide which one I would like to invest more time in. Basic hello >> world benchmark can be one of the factor that will affect my decision. >> So if aleph community wants to grow matching (or beating) competitors >> in basic benchmark may help.
> I don't see this regression in performance. I'm using the latest snapshots > of aleph 0.1.3 and lamina 0.3.1. > I think it's good to hear this kind of feedback from people interested in > Aleph. I agree that Aleph should at least be competitive w/ Node.js on this > baseline. > Zach, do you have any thoughts on this? > David
