Hello. We are heavy users of Mongo 2.2 replica sets. I've noticed that for some replica sets, certain nodes will be heavily accessed (loadavg > 40) while at the same time other nodes will be under utilized. I've experimented with removing the "busy" node from the replica set and adding it back to distribute the workload more evenly. My hunch is that the worker processes accessing the replica sets "bind" themselves to a particular node and stay "bound" to that node even if it is super busy.
This is probably not a Mongo issue, but we need a solution to more evenly balance our workload. We're considering writing a layer in front of the replica set, possibly using ZooKeeper.
I would appreciate any ideas or recommendations. Thanks in advance.