Implosion

[martin_clausen]

Inspired by this post : http://dosync.posterous.com/22397098 I tried
benchmarking Aleph on my two machines: a MacBook 2.4 Ghz Intel Core
Duo and a Dell Studio XPS Intel Core i7 2.8 Ghz running Ubuntu 10.04.

I ran the hello-world example described here
http://wiki.github.com/rje/aleph/hello-world-with-aleph.

I get the expected "Hello world!" response in my browser when visiting
127.0.0.1:8080 and the same using curl from the command line.

I tried benchmarking the code with Apache Bench using the following
command:

ab -n 5000 -c 50 http://127.0.0.1:8080/

On both machines this results in (after a bit of waiting):

apr_poll: The timeout specified has expired (70007)

So I tried siege instead using:

siege -b -r 5000 -c 50 localhost:8080/

Here the result is slightly more interesting. On the MacBook the
bechmark runs a while (30-40 secs) and then spits out a string of:

warning: socket: -1341648896 select timed out: Operation timed out

When I break with ^C I get:

Lifting the server siege... done.
Transactions: 0 hits
Availability: 0.00 %
Elapsed time: 42.99 secs
Data transferred: 0.00 MB
Response time: 0.00 secs
Transaction rate: 0.00 trans/sec
Throughput: 0.00 MB/sec
Concurrency: 0.00
Successful transactions: 0
Failed transactions: 50
Longest transaction: 18588871729648130931857277272981504.00
Shortest transaction: 0.00

On the Dell doing the same I get a string of successful transactions
and then:

Error: socket: -350578928 address is unavailable.: Cannot assign
requested address
done.

Followed by:

siege aborted due to excessive socket failure; you
can change the failure threshold in $HOME/.siegerc
Transactions: 28080 hits
Availability: 96.32 %
Elapsed time: 10.68 secs
Data transferred: 0.67 MB
Response time: 0.02 secs
Transaction rate: 2629.21 trans/sec
Throughput: 0.06 MB/sec
Concurrency: 44.25
Successful transactions: 28080
Failed transactions: 1073
Longest transaction: 0.95
Shortest transaction: 0.00

Any idea what is going on?

[David Nolen]

On Sun, Aug 1, 2010 at 10:44 AM, martin_clausen <martin....@gmail.com> wrote:

Any idea what is going on?

I think this has something to do with how fast the OS can deliver and release file descriptors. This affects node.js as well. You could tweak your system to allow for more open files. Or if you just decrease the number concurrent clients ab should work just fine.

David

[Zach Tellman]

Nope, this was a bug I introduced w.r.t. closing connections.  My tests didn't capture this regression, sorry.

It's worth pointing out that currently Aleph isn't as fast as when David was running his benchmarks.  About 1% of responses are 100x slower than average, which greatly reduces the average RPS.  I'm looking into this.

Zach

[Zach Tellman]

Okay, the performance issue was an issue with transactions.  I've put in a workaround until I can spend more time on it, so everything should be working fine now.  Let me know if you find any further issues.

Zach

[Martin Clausen]

Is clojars updated, or is it necessary to pull from github?

/mac

[ztellman]

You can pull from clojars.

On Aug 2, 9:26 am, Martin Clausen <martin.clau...@gmail.com> wrote:
> Is clojars updated, or is it necessary to pull from github?
>
> /mac
>
>
>
> On Mon, Aug 2, 2010 at 12:46 AM, Zach Tellman <ztell...@gmail.com> wrote:
> > Okay, the performance issue was an issue with transactions.  I've put in a
> > workaround until I can spend more time on it, so everything should be
> > working fine now.  Let me know if you find any further issues.
> > Zach
>

> > On Mon, Aug 2, 2010 at 12:08 AM, Zach Tellman <ztell...@gmail.com> wrote:
>
> >> Nope, this was a bug I introduced w.r.t. closing connections.  My tests
> >> didn't capture this regression, sorry.
> >> It's worth pointing out that currently Aleph isn't as fast as when David
> >> was running his benchmarks.  About 1% of responses are 100x slower than
> >> average, which greatly reduces the average RPS.  I'm looking into this.
> >> Zach
>

> >> On Sun, Aug 1, 2010 at 11:36 PM, David Nolen <dnolen.li...@gmail.com>

[martin_clausen]

The new version works fine with:

ab -n 250000 -c 50 http://localhost:8080/

I get around 13.000 requests/sec.

siege -b -r5000 -c50 localhost:8080/

still dies with:

siege aborted due to excessive socket failure; you
can change the failure threshold in $HOME/.siegerc

Transactions: 102995 hits
Availability: 98.99 %
Elapsed time: 13.20 secs
Data transferred: 2.46 MB
Response time: 0.01 secs
Transaction rate: 7802.65 trans/sec
Throughput: 0.19 MB/sec
Concurrency: 43.85
Successful transactions: 102995
Failed transactions: 1054
Longest transaction: 0.16
Shortest transaction: 0.00

[ztellman]

I'm not able to reproduce your issue. I've added a default settings
for the server that might help, though. Can you repull from clojars
and tell me if that changes anything?

Zach

On Aug 6, 11:15 pm, martin_clausen <martin.clau...@gmail.com> wrote:
> The new version works fine with:
>

> ab -n 250000 -c 50http://localhost:8080/

[Martin Clausen]

It still dies after approx. 100K hits. I am running 64bit Ubuntu, but
that shouldn't make a difference, right?

/mac

[Zach Tellman]

Shot in the dark: can you try 'echo 1 >
/proc/sys/net/ipv4/tcp_tw_reuse' and then rerun it? On my OS X
installation, I was getting long stalls every 16384 requests because
each ephemeral port could only be used once every 15 seconds or so.
If you're having a similar problem on your Linux install, I think that
will fix it.

Zach

[Martin Clausen]

It was set at 0. Setting to 1 solves the issue. Thanks.

/mac

[David Nolen]

Curious, what's the best way to fix this on OS X ?

On Sat, Aug 7, 2010 at 3:16 PM, Zach Tellman <ztel...@gmail.com> wrote:

/proc/sys/net/ipv4/tcp_tw_reuse

[Martin Clausen]

I have no clue. Only made it work on the Ubuntu box. Doesn't look like
there is an OS option to control this on OS X.

/mac

[Zach Tellman]

http://www.brianp.net/2008/10/03/changing-the-length-of-the-time_wait-state-on-mac-os-x/