Unexpected behavior when mongos is bounced

[Brent Wilkins]

We have a version 3.6.3 mongo 2X3 (2 shards, each with 3 relica nodes)  QA cluster. There are 5 mongos instances available. We are using version 0.13.0 of the driver. We connect via a uri connection string like the following:

"mongodb://user:password@mongos-1:2017,mongos-2:2017/admin?rm.keepAlive=true"

Let us assume at the time we connect, we connect via mongos-1. All database operations work as expected. Now we bounce mongos-1 (for maintenance or whatever). We expect that we will automatically be connected to the cluster via mongos-2. However, we receive the following error from the reactivemongo driver:

"MongoError['No active channel with 'SecondaryPreferred()' found for the nodes: 'mongos-1:27017' [Primary] { authenticated:10, connected:10, channels:10 }, 'mongos-2:27017' [Unknown] { authenticated:0, connected:0, channels:0 } (Supervisor-1/Connection-2)']"

We continually get this error until we bounce our service or otherwise restart, even after mongos-1 is online after the bounce. After some experimentation, we found that if we remove the "rm.keepAlive=true" from the connection uri, every thing works as expected (automatic connection via mongos-2). We originally added the "rm.keepAlive=true" as the mongos instances were load balanced behind haproxy. We needed to keep the connection busy or else haproxy would close it. We now either directly list the mongos hosts in the connection uri (as in this example) or use a VIP which supports load balancing with stickiness. Thus we do not need to specify keepAlive.

Can you explain this driver behavior? It doesn't seem correct to me.

Thanks,

Brent

[Cédric Chantepie]

Would require a reproducer to be able to investigate (Docker usage recommended).