OOM on secondary re-sync, bgsync buffer consumes all ram and swap

[Brent Miller]

We're running into this issue where we cannot re-sync our secondaries after taking them off line for more than 10 minutes or so. When we take try to bring a secondary back on-line it will consume all ram and swap and then crash.

When we start mongod with -vv, the log files on the crashed secondary is filled with:

Thu Oct 11 12:38:22 [rsBackgroundSync] bgsync buffer has 5171004922 bytes

Thu Oct 11 12:38:22 [rsBackgroundSync] bgsync buffer has 5171152019 bytes

Thu Oct 11 12:38:22 [rsBackgroundSync] bgsync buffer has 5171257666 bytes

Thu Oct 11 12:38:22 [rsBackgroundSync] bgsync buffer has 5171402953 bytes

Thu Oct 11 12:38:22 [rsBackgroundSync] bgsync buffer has 5171534548 bytes

Thu Oct 11 12:38:22 [rsBackgroundSync] bgsync buffer has 5171644793 bytes

Thu Oct 11 12:38:22 [rsBackgroundSync] bgsync buffer has 5171751196 bytes

Thu Oct 11 12:38:22 [rsBackgroundSync] bgsync buffer has 5171896415 bytes

Thu Oct 11 12:38:22 [rsBackgroundSync] bgsync buffer has 5172027296 bytes

Thu Oct 11 12:38:23 [rsBackgroundSync] bgsync buffer has 5172138255 bytes

The primary is running 2.2.0, Ubuntu 12.04 with 48G ram. We have two secondaries that are running 2.2.0, Ubuntu 12.04, but only have 4G of ram. (We see the same behavior when running 2.2.1rc0 too.)

Is there a reason why the bgsync buffer is growing so large? Is there a way to minimize this?

The rs.config() is as follows:

{

"_id" : "apm",

"version" : 27,

"members" : [

{

"_id" : 0,

"host" : "apc:27017",

"priority" : 2

},

{

"_id" : 3,

"host" : "apm-dev:27017"

},

{

"_id" : 4,

"host" : "apm:27017"

}

]

}

Thanks,

Brent

[Dwight Merriman]

are you running 64 bit or 32 bit?

did you configure --oplogsize on the command line (the default behaves well usually so the question is mainly if it is non-default)?

if you can post rs.status() that would be helpful.

[Brent Miller]

All machines are 64bit, and yeah, we've configured the oplog manually to 32G (oplogSize = 32768) This was due to the fact that when we first started testing, we only partitioned out 256G of disk for Mongo, and found that the oplog ended up being too small to add new secondaries. (It might be worth noting that we have a fairly write-heavy load and we're making a fair number of updates to documents that are on the order of a megabyte.)

In order to keep the replica set up, I've had to demote one of the secondaries to be an arbiter and just disabled the other, so I don't know if rs.status() would be any help at the moment. I can start a re-sync later on today if it will be helpful. However, once we get the secondaries through their inital sync, their oplog will trail the primary anywhere from 1 - 60 seconds.

Thanks,

Brent

[Dwight Merriman]

please run this in the shell and post the result?

db.printReplicationInfo()