Vert.x performance tests

[Gianluca Lupo]

Hi all,

I'm trying to get benchmarks about performance of my vertx (2.1RC3) server based application.

Server-side: using vertx I'm publishing some RESTful services to be used by the client layer and in the same vertx instance there is also a verticle to manage websocket communication and exchange small json messages  with client layer

Client-side: a web application which calls the RESTful services and also a web socket in order to receive messages from server (using SockJS).

The infrastructure is the following:

browser <------> NGINX -> VERTX -> REDIS

I developed a client application in order to have performance test benchmarks and my choise is to use a vertx instance in my client to test the server. Every communication is done in async way, so also the calls to restful services. Once subscribed the web socket I keep it alive so I can exchange an arbitraty number of messages. So I store httpclient and websocket for each connection in my client in a specific data structure.

My goal is to get a maximun number of connection which I can open to websocket communication layer, because I want to check this aspect of the application server side: I would like to know how many alive connection my server can handle.

Everything works fine except when I reach more than 6000/7000 connection (client side) when I start getting this exception:

apr 09, 2014 10:08:36 AM org.vertx.java.core.logging.impl.JULLogDelegate error

Grave: Unhandled exception

java.net.BindException: Address already in use: no further information

        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)

        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)

        at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:191)

        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:279)

        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)

        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:461)

        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:378)

        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:350)

        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)

        at java.lang.Thread.run(Thread.java:744)

No exceptions server side.

It seems the OS (currently my client is running on Windows 7 64 bit) is not able to handle more TCP connections. I'm investigating on it, maybe on OS limits, but any help and idea is welcome.

Thanks

G.

[Norman Maurer]

Yes sounds like an OS issue, unfortunately I’m not using windows at all and so have no idea what you need to adjust :/

-- 
Norman Maurer

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Tim Fox]

On 09/04/14 09:33, Norman Maurer wrote:

Yes sounds like an OS issue, unfortunately I’m not using windows at all and so have no idea what you need to adjust :/

What Norman said, but a bit more detail...

Firstly you probably need to increase the maximum number of connections / file handles in your OS settings. I've no idea how to do this for Windows (maybe google it?) but for Linux it's described in the Vert.x docs.

Secondly, if you're opening and closing connections rapidly you will eventually run out of free address/port if you're not re-using addresses. By default SO_REUSEADDRESS is probably false, and the way TCP works is, even after you've closed a connection it remains in a TIME_WAIT state for as while longer. More info here http://stackoverflow.com/questions/3229860/what-is-the-meaning-of-so-reuseaddr-setsockopt-option-linux - Windows has a similar setting.

You should be able to set client.setReuseAddress(true) to enable reuse. You'll also need to to do this on the server side.

There are various other tunings that should be done if you want to handle many connections which, again, are OS specific.

In summary I wouldn't recommend doing your own benchmarking as it is so easy to come up with a broken benchmark that isn't really testing the SUT ;)

[petermd]

some info here

http://www.speedguide.net/articles/windows-7-vista-2008-tweaks-2574

MaxUserPort is setting for the number of ports you can have per IP, but as Tim mentions if your tester is opening/closing lots of connections you might need to shorten TcpTimedWaitDelay also so that the OS can re-use ports.

-Peter

[Gianluca Lupo]

Thanks a lot. 

I'm just investigating about addresses reuse and I will try it soon. 

G.

[Venkat]

 @ Lupo,

  could you able to find out the resolution...??
  let us know the progress as we are also into same kind of trouble..
  Thanks...

[Gianluca Lupo]

Hi Srikant,

there is no "the resolution" for the problem.

First of all I modified the code in order to be sure that I reuse addresses on http client and some other small improvements.

Secondly I tuned windows registry socket setting to decrease the timeout when the socket is in state TIME_AWAIT and this helped very much, secondly I increased the maximum number of allowed socket available and also this tune has increased the results.

In any case testing the same application on a Unix-based machine, setting ulimit to maximum number, gets completely different results.

Just last note before running test cases one after another i verified that there where no pending ports still opened from previous session tests.

If you need further information please let me know.

G.

--
You received this message because you are subscribed to a topic in the Google Groups "vert.x" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/vertx/UJbd5ByvJ_Q/unsubscribe.
To unsubscribe from this group and all its topics, send an email to vertx+un...@googlegroups.com.