Mongo High Replication Lag

[Nimi Wariboko Jr]

5 days ago we brought started added a replica set (and arbiter) to our single node replica sets (That had about ~400GB of data each). 4 days later they were finally elevated to secondaries. Our plan was to make the new secondaries primaries, however that isn't possible because our secondaries are over 24 hours behind and it doesn't seem they are catching up (I have been tracking them, and after the initial indexes were built, the machines seemed to be perpetually 24 hours behind).

We insert about 5 Gigabytes of data/day, with most of it coming in at about 12 - 6 AM.

Here is the replication status:

Set 1:

{

"set" : "RS1",

"date" : ISODate("2012-09-25T22:42:08Z"),

"myState" : 1,

"members" : [

{

"_id" : 0,

"name" : "dbs1d1:27018",

"health" : 1,

"state" : 1,

"stateStr" : "PRIMARY",

"uptime" : 1556401,

"optime" : Timestamp(1348612869000, 1),

"optimeDate" : ISODate("2012-09-25T22:41:09Z"),

"self" : true

},

{

"_id" : 1,

"name" : "rs1_arb:28017",

"health" : 1,

"state" : 7,

"stateStr" : "ARBITER",

"uptime" : 15801,

"lastHeartbeat" : ISODate("2012-09-25T22:42:07Z"),

"pingMs" : 0

},

{

"_id" : 2,

"name" : "dbs1b:27018",

"health" : 1,

"state" : 2,

"stateStr" : "SECONDARY",

"uptime" : 2814,

"optime" : Timestamp(1348519459000, 43),

"optimeDate" : ISODate("2012-09-24T20:44:19Z"),

"lastHeartbeat" : ISODate("2012-09-25T22:42:07Z"),

"pingMs" : 1

}

],

"ok" : 1

}

{

"set" : "RS2",

"date" : ISODate("2012-09-25T22:43:05Z"),

"myState" : 1,

"members" : [

{

"_id" : 0,

"name" : "dbs1d2:27018",

"health" : 1,

"state" : 1,

"stateStr" : "PRIMARY",

"uptime" : 1302813,

"optime" : Timestamp(1348610808000, 2),

"optimeDate" : ISODate("2012-09-25T22:06:48Z"),

"self" : true

},

{

"_id" : 1,

"name" : "rs2_arb:28018",

"health" : 1,

"state" : 7,

"stateStr" : "ARBITER",

"uptime" : 15829,

"lastHeartbeat" : ISODate("2012-09-25T22:43:03Z"),

"pingMs" : 1

},

{

"_id" : 2,

"name" : "dbs2b:27018",

"health" : 1,

"state" : 2,

"stateStr" : "SECONDARY",

"uptime" : 42042,

"optime" : Timestamp(1348486720000, 202),

"optimeDate" : ISODate("2012-09-24T11:38:40Z"),

"lastHeartbeat" : ISODate("2012-09-25T22:43:03Z"),

"pingMs" : 0

}

],

"ok" : 1

}

Are there any options to speed up the sync? We really want to avoid having to take the primaries down.

[Nimi Wariboko Jr]

Anyone wondering how we eventually solved this, we moved the secondaries to SSDs.

[Octavian Covalschi]

Thanks for the info. Was it a any kind of RAID setup ?

--
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com
To unsubscribe from this group, send email to
mongodb-user...@googlegroups.com
See also the IRC channel -- freenode.net#mongodb