Freezing during apache bench load test

[Isaac Schlueter]

I'm finding that a simple hello world server is freezing up during ab
load testing. It seems to usually occur around 16,000 requests, but
the exact number varies. It freezes up for a second or two and won't
serve any requests, and then seems to "wake up" and start responding
again.

For a while I was thinking that it was ab acting up. But curl also
doesn't get a response for several seconds when it's in this state.

Here's the server that I'm running:

require("http").createServer(function (_, response) {
response.writeHead(200, {});
response.write("hello");
response.close();
}).listen(8000);

And the ab command:

ab -k -c 50 -n 20000 http://0.0.0.0:8000/

Also tried this:

ab -k -c 50 -n 20000 http://127.0.0.1:8000/

Same results on HEAD and v0.1.33. Omitting the -c 50 seems to have no
significant effect.

Any ideas?

--i

[Erik Corry]

GC?

Try running with --trace-gc

You might also try --never-compact though the long term consequences
of that are unknown (the heap may stabilize or it may get more and
more fragmented and grow).

2010/3/26 Isaac Schlueter <i...@izs.me>:

> --
> You received this message because you are subscribed to the Google Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com.
> To unsubscribe from this group, send email to nodejs+un...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/nodejs?hl=en.
>
>

[Jorge]

On Mar 26, 9:37 am, Isaac Schlueter <i...@izs.me> wrote:
> I'm finding that a simple hello world server is freezing up during ab
> load testing.  It seems to usually occur around 16,000 requests, but
> the exact number varies.  It freezes up for a second or two and won't
> serve any requests, and then seems to "wake up" and start responding
> again.
>
> For a while I was thinking that it was ab acting up.  But curl also
> doesn't get a response for several seconds when it's in this state.
>
> Here's the server that I'm running:
>
> require("http").createServer(function (_, response) {
>   response.writeHead(200, {});
>   response.write("hello");
>   response.close();
>
> }).listen(8000);
>
> And the ab command:
>

> ab -k -c 50 -n 20000http://0.0.0.0:8000/
>
> Also tried this:
>
> ab -k -c 50 -n 20000http://127.0.0.1:8000/

>
> Same results on HEAD and v0.1.33.  Omitting the -c 50 seems to have no
> significant effect.
>
> Any ideas?

The same is happening to me:
http://groups.google.com/group/nodejs/browse_thread/thread/e4c8ca9430dafc53#5acb52814507c99f

On OSX 10.6.2. What OS is yours ?

I have opened a separate shell with a netstat -w 1 to monitor the
traffic, and there are cyclic freezes:

$ netstat -w 1
input (Total) output
packets errs bytes packets errs bytes colls
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
1 0 125 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
1 0 112 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
2 0 106 1 0 46 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
input (Total) output
packets errs bytes packets errs bytes colls
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
22289 0 2029184 22289 0 2029170 0
43920 0 3992248 43920 0 3992248 0
43932 0 3994074 43932 0 3994074 0
43664 0 3970829 43664 0 3970829 0
43711 0 3973479 43711 0 3973479 0
43469 0 3954326 43469 0 3954326 0
1146 0 106110 1145 0 105985 0
3 0 168 3 0 168 0
3 0 168 3 0 168 0
3 0 168 3 0 168 0
3 0 168 3 0 168 0
3 0 168 3 0 168 0
1 0 112 0 0 0 0
3 0 168 3 0 168 0
input (Total) output
packets errs bytes packets errs bytes colls
0 0 0 0 0 0 0
1 0 60 1 0 46 0
1 0 46 0 0 0 0
3 0 152 3 0 152 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
3 0 152 3 0 152 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
2 0 106 1 0 46 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0
1 0 85 2 0 171 0
input (Total) output

OTOH, if while it's in the frozen state I do in another shell a
netstat -f inet and I see that there are lots (several thousands) of
ports in TIME_WAIT, IOW, ISTM, the ports are not being released
quickly enough:

$ netstat -f inet
Active Internet connections
Proto Recv-Q Send-Q Local Address Foreign Address
(state)
tcp4 0 0 localhost.http localhost.53454
TIME_WAIT
tcp4 0 0 localhost.http localhost.53454
TIME_WAIT
.
.
tcp4 0 0 localhost.http localhost.60456
TIME_WAIT
tcp4 0 0 localhost.http localhost.60457
TIME_WAIT

$ netstat -f inet | grep TIME_WAIT | wc -l
16379

And, it happens as well with the Apache server -IT'S NOT ONLY NODE !-,
so it must be an OS issue, or an ab issue, or there's a valid reason
for this that we ignore.. :-)

Any thoughts ?
--
Jorge.

[Ray Morgan]

I have seen a lot of "pausing" with ab on Mac OSX. This happens with
pretty much every local server I have tested against. Not sure why but
it makes me believe it is an OS issue.

Can anyone reproduce this on a non Mac?

-Ray

[Scott González]

http://stackoverflow.com/questions/1216267/ab-program-freezes-after-lots-of-requests-why

Maybe we should explain this somewhere on the node docs since someone seems to ask about this every few weeks.

[Jorge]

On Mar 26, 4:44 pm, Scott González <scott.gonza...@gmail.com> wrote:
>
> http://stackoverflow.com/questions/1216267/ab-program-freezes-after-l...

As clear as an unmuddied lake, Fred. As clear as an azure sky of
deepest summer.

> Maybe we should explain this somewhere on the node docs since someone seems
> to ask about this every few weeks.

Yes...

Thanks,
--
Jorge.

[Isaac Schlueter]

Ah, I knew I'd seen this talked about before. Thanks, guys.

I wonder why it never affected my mac until just now. Weird.

--i

[Dav Glass]

I've been using seige and it seems to perform a little better than ab:
http://www.joedog.org/index/siege-home

--
Dav Glass
davg...@gmail.com
blog.davglass.com

+ Windows: n. - The most successful computer virus, ever. +
+ A computer without a Microsoft operating system is like a dog
without bricks tied to its head +
+ A Microsoft Certified Systems Engineer is to computing what a
McDonalds Certified Food Specialist is to fine cuisine +

[Jorge]

On 26/03/2010, at 16:44, Scott González wrote:

http://stackoverflow.com/questions/1216267/ab-program-freezes-after-lots-of-requests-why

Maybe we should explain this somewhere on the node docs since someone seems to ask about this every few weeks.

JFTR:

http://en.wikipedia.org/wiki/Ephemeral_port

To see the current settings:

$ sysctl -a | grep ip.portrange

->

net.inet.ip.portrange.lowfirst: 1023

net.inet.ip.portrange.lowlast: 600

net.inet.ip.portrange.first: 49152

net.inet.ip.portrange.last: 65535

net.inet.ip.portrange.hifirst: 49152

net.inet.ip.portrange.hilast: 65535

To modify it:

$ sudo sysctl -w net.inet.ip.portrange.first=32768

$ sudo sysctl -w net.inet.ip.portrange.hifirst=32768

->

net.inet.ip.portrange.first: 49152 -> 32768

-- 

Jorge.

[Santi Manninen]

Hi!

Just doing this same test myself on Mac with 0.1.96. I have applied
the sysctl changes, but still seeing timeouts and 'Connection reset by
peer'.

$ ab -k -c 250 -n 10000 http://127.0.0.1:8000/
[...]
Completed 4000 requests

Test aborted after 10 failures

apr_socket_connect(): Connection reset by peer (54)

$ sysctl -a | grep ip.portrange

net.inet.ip.portrange.lowfirst: 1023
net.inet.ip.portrange.lowlast: 600
net.inet.ip.portrange.first: 32768
net.inet.ip.portrange.last: 65535
net.inet.ip.portrange.hifirst: 32768
net.inet.ip.portrange.hilast: 65535

Best regards,

Santi

On Apr 8, 2:31 pm, Jorge <jo...@jorgechamorro.com> wrote:
> On 26/03/2010, at 16:44, Scott González wrote:
>

> >http://stackoverflow.com/questions/1216267/ab-program-freezes-after-l...