HttpClient burst request handling, No SYN packets seen in wireshark for timed out connections

[Mehar]

Hi,

I have a following scenario in my application.

My app has to talk to 5K devices over https, I posted independent test code that you can run.

You can run the code like below 

ulimit -n 1000000

java -DhostCount=5000 -cp client.jar VertxHttpClient 80.1.1.1 443 /iox 5000 1000 1 2>&1 | tee out.log

this code is going to generate 1 request for each of 5K hosts starting from 80.1.1.1

Out of 5K requests some 2K will succeed and rest of them fail with connection timed out.

java.net.ConnectException: Connection timed out: /80.1.15.75:443

at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:1.8.0_66]

at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:1.8.0_66]

at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:224) ~[netty-transport-4.0.33.Final.jar:4.0.33.Final]

at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:289) [netty-transport-4.0.33.Final.jar:4.0.33.Final]

at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:528) [netty-transport-4.0.33.Final.jar:4.0.33.Final]

at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) [netty-transport-4.0.33.Final.jar:4.0.33.Final]

at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) [netty-transport-4.0.33.Final.jar:4.0.33.Final]

at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) [netty-transport-4.0.33.Final.jar:4.0.33.Final]

at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112) [netty-common-4.0.33.Final.jar:4.0.33.Final]

at java.lang.Thread.run(Thread.java:745) [?:1.8.0_66]

During this time my server resources are free and it is able to respond to request using a different client with smaller rate of request, So the isn't overloaded.

Another observation is that for connections timed out, I don't see "SYN" packets on the tcpdump as well.

I wanted to confirm that my server isn't the problem, I ran the following tests from the same client machine (https://github.com/tsenart/vegeta)

./vegeta attack -targets targets.txt -insecure -rate 5000 -duration 5s | tee results.bin | ./vegeta report

This client successfully gets responses for all the 5K requests.

Hope the information provided is sufficient, will be happy to provide any other information required.

Appreciate any help regarding this issue.

Thanks

-Mehar