io.vertx.core.impl.NoStackTraceThrowable: ERR max number of clients reached

[E. Ulrich Kriegel]

Hi there, 

i stress my verticles with gatling, and after about 2000 sec we  get  the following error Message:

io.vertx.core.impl.NoStackTraceThrowable: ERR max number of clients reached

Since i can get no stack trace, it's hard to find the source of that error.

the corresponding verticle accesses a Postgres database and a redis server.

The error occurs in Vertx 3.3.3 and 3.4.0

Thanks in advance

--Ulrich

[Paulo Lopes]

what is your your postgres max_connections value? and the connection pool max size?

Could it be that you're hitting the pool/db limits?


[Julien Viet]

it is a redis error.

-- 
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/5dd86326-8b29-4be5-bb6a-71edaeab7024%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Paulo Lopes]

If its a Redis error then there is something wrong, you should not need more than 1 client per event loop. Can you provide a reproducer?

[E. Ulrich Kriegel]

Hi Paulo and Julian,

i increased max_pool_size to 50 and initial pool size to 30, but the error still occurs.

The problem only occurs after ~ 15 min, if 8 user/per second continously access the service in parallel.

I will try to get rid of the database and simplify the code, but you will need to use Gatling to reproduce the error.

May be it already helps to see, how the redicClient is created

public class DisposableRedisConnection {

private RedisClient redisClient = null;

public Single<RedisClient> create(Vertx vertx, JsonObject conf, Logger logger){

try{

redisClient = RedisClient.create(vertx, new RedisOptions(conf));

}catch(Throwable t){

System.out.println("DisposableRedisConnection: "+t);

logger.error(new LoggerObject(this.getClass().getName()).setErrorStatus(t).setSubject(this.getClass().getName()));

}

return Single.just(redisClient);

}

public void dispose(){

if(redisClient != null){

redisClient.rxClose();

redisClient = null;

}

}

}

Single.using(DisposableRedisConnection::new, f -> f.create(vertx, redisConfigArray.getJsonObject(0), logger), f -> f.dispose()).subscribe( redisClient->{ ...}

[Paulo Lopes]

There are no pools in Redis. What do you mean that you increased it to 50?

Sorry, we need a reproducer otherwise I've no idea what is going on.

[E. Ulrich Kriegel]

The pools are the answer to your question concerning the postgres_max_connection_value.

Am Dienstag, 14. März 2017 09:17:25 UTC+1 schrieb Paulo Lopes:

There are no pools in Redis. What do you mean that you increased it to 50?

Sorry, we need a reproducer otherwise I've no idea what is going on.

I'm working on it. 

[Tim Fox]

Are you creating a new RedIsClient and closing it for every command you execute?

Each RedisClient maintains a connection to the server and is designed to be reused between commands. Perhaps you're running out of connections on the server side due to the large number created at any one time? Also creating a new one each time is going to be slow.

[E. Ulrich Kriegel]

Each httpHandler has its own redis Client, which is closed after the handler is ready.

I'm preparing a reproducer.

[Tim Fox]

If you could remove any usage of rx from the reproducer that would be helpful too.

[E. Ulrich Kriegel]

There is a reproducer (rx-based) as Eclipse-project and a gatling test scenario available via https://owncloud.fokus.fraunhofer.de/index.php/s/UDxIpdNC5cp4AV8

Build the test verticle in the project directory

mvn clean install

and start it with from the projects directory

java -Xmx4g -Xms4g -jar target/redistest.0.0-fat.jar  -conf src/main/resources/config/config-fr-postgres-new.json

Unpack the gatling archive, set the env-variable GATLING_HOME to the location of the unpacked GATLING bundle  and start the simulation

sh $GATLING_HOME/bin/gatling.sh
 

After 3 CR the simulation starts. On a MacBoocPro with 16GBRAM it takes approx. 350 sec until the error occurs

[Tim Fox]

Can you push it to GitHub? It makes it a lot easier to view, reference lines of code, comment etc :)

[Tim Fox]

It seems the DoInterestingThings.doIt() method creates a new RedisClient each time, and this is called on every search request. This would be problematic as mentioned in a previous reply.

On Tuesday, 14 March 2017 13:53:39 UTC, E. Ulrich Kriegel wrote:

[E. Ulrich Kriegel]

That means,

the start method is a good place to create the client and the stop method closes it afterwards.

Thanks for the hint.

[E. Ulrich Kriegel]

  Its now under 

            https://github.com/ukr15/vertx-redis                  

[Tim Fox]

+1