Time out waiting connection from connection pool
Роман Степанов
Hello!
I have strange problem on production. At some unknown reason my application receives timeout exception while waiting connection to mongodb from pool. But there is no connection loss between app server and mongo server. All servers and apps are available at this moment, no ethernet loss or other sort of unavailability. No errors or problems in logs of mongo. This sort of "connection" problem take place for only a few seconds. But in my case, 1100 - 1200 rps, such blinking causes a huge problem.
Can You please help me solve this problem or at least direct me where to look?
My setup:
Mongo Cluster - 7 servers (10 CPU, 32 RAM, OS RHL8):
- 1 muster
- 1 arbiter
- 5 slaves
App – 4 servers (4 CPU, 8 RAM, OS RHL8). Each application is independent.
All requests come though Citrix Load Balancer, with round robin politics.
All servers work on VMWare.
KMongo version – 4.7.1
Java version: openjdk version "1.8.0_332", OpenJDK Runtime Environment (build 1.8.0_332-b09), OpenJDK 64-Bit Server VM (build 25.332-b09, mixed mode
Kotlin version – 1.6.10
MongoDb version – 4.4.14 community
Connection settings:
val mongoOptions = MongoClientSettings.builder().applicationName("Info")
.applyToClusterSettings {
it.hosts(seeds)
it.mode(ClusterConnectionMode.MULTIPLE)
it.requiredClusterType(ClusterType.REPLICA_SET)
it.serverSelectionTimeout(10_000L, TimeUnit.MILLISECONDS)
}
.applyToConnectionPoolSettings {
it.maxConnectionIdleTime(60000, TimeUnit.MILLISECONDS)
it.maxConnectionLifeTime(900000, TimeUnit.MILLISECONDS)
it.maxSize(3000)
it.minSize(20)
it.maxWaitTime(500, TimeUnit.MILLISECONDS)
}
.readPreference(ReadPreference.secondaryPreferred())
.readConcern(ReadConcern.AVAILABLE)
.writeConcern(WriteConcern.W1)
.retryWrites(true)
.retryReads(true)
.credential(cred)
.build()
val mongoConnection = KMongo.createClient(ConnectionString(config.getMongoDBConfig().connectionString))
