RequestTimeout issues

[Edi V]

Hi All,

We are currently using AHC version 2.5.4 & noticing the following behavior with respect to timeouts specifically

when setting the below configs  -

max_con = 10000;

.setMaxConnections(max_con)

.setRequestTimeout(3000)

.addRequestFilter(new ThrottleRequestFilter(max_con))

We decided to use ThrottleRequestFilter to throttle requests when we hit the max_con limit.

However I noticed that whenever the request times out we get org.asynchttpclient.exception.TooManyConnectionsException. 

I tried to dig into the AHC code (specifically ThrottleRequestFilter & MaxConnectionSemaphore) & noticed that they both use different semaphores to

throttle. I am suspecting whether this is causing the TooManyConnectionsException to be thrown specifically when the request times out.

I do not see this exception being thrown when I don't set any timeout for the requests. However our requirement is to support timeout.

I also noticed that few days back support for BlockingConnectionSemaphoreFactory was added in the following PR -

https://github.com/AsyncHttpClient/async-http-client/pull/1586

So I pulled the latest AHC and modified my code as follows to use setAcquireFreeChannelTimeout & removed ThrottleRequestFilter.

max_con = 10000;

.setMaxConnections(max_con)

.setRequestTimeout(3000)

.setAcquireFreeChannelTimeout(30000)

Now when I run my test with some timeouts I do not see TooManyConnectionsException being thrown. However after a while I do see 

"java.net.SocketException: Too many open files". 

The ulimit -n is 100000. I am not sure whether this is an expected behavior.

Regards,

Aditya

[mkoz...@gmail.com]

Hi.

Could you check in /proc/PID/limits if limits are really applied? I had once that ulomit -n was not enough and changes were needed in /etc/security/limits.conf.

Best regards,

Michal

--
You received this message because you are subscribed to the Google Groups "asynchttpclient" group.
To unsubscribe from this group and stop receiving emails from it, send an email to asynchttpclie...@googlegroups.com.
To post to this group, send email to asyncht...@googlegroups.com.
Visit this group at https://groups.google.com/group/asynchttpclient.
For more options, visit https://groups.google.com/d/optout.

[Edi V]

Hi Michal,

As you mentioned the process level limits were not set correctly. Now I have updated the limit (100000) and its reflected properly at the process level.

Even now with the following settings -

asyncHttpClient = Dsl.asyncHttpClient(Dsl.config().setThreadPoolName("async-tp")
            .addRequestFilter(new ThrottleRequestFilter(10000))
            .setRequestTimeout(3000)
            .setMaxConnections(10000)
            .setIoThreadsCount(iot)
            .setMaxConnectionsPerHost(10000)
            .setHandshakeTimeout(60000)
            .setConnectTimeout(60000)
            .setKeepAlive(true));

I still see the following exception -

org.asynchttpclient.exception.TooManyConnectionsException: Too many connections: 10000

Regards,

Aditya

[Edi V]

Also I am using the following -

openjdk version "1.8.0_191"

AHC - 2.5.4

Regards,

Aditya

[Edi V]

Looks like the problem is with the number of io threads I am specifying for 10k max connections/ThrottleRequestFilter for p99 response time of 400 ms. Will share more my findings once I have a handle on the issue.

Regards,

Aditya

[Edi V]

I am currently using AHC 2.5.4 with - 

4 io threads

1 response thread (All it does it checks status_code=200 or not)

max_conn = 11k

ThrottleRequestFilter=10k

I have a 16 core instance and I see cpu util stuck at 1200%. I am not sure what is causing the issue.

I have shared the thread dump here -

https://gist.github.com/advaidya/b3a771c82bed99c1529af3409616bc2c

Regards,

edi

[Stéphane LANDELLE]

I'm not fond of this ThrottleRequestFilter. You're most likely throwing requests at AHC without considering what happens when you exceed your throttle.

There's a very good chance you end up with a humongous number of requests buffered in memory.

You should consider something like reactive-streams or RxJava to deal with back pressure, ie slow down the producer when the consumer is saturated.

Stéphane Landelle

GatlingCorp CTO

slan...@gatling.io

[Edi V]

Thanks stephane.

Regards 

Aditya