Replica stuck at STARTUP, bi-directional mongo comms work fine

17 views

Skip to first unread message

James Green

unread,

Aug 26, 2016, 5:40:26 AM8/26/16

to mongodb-user

Here's our primary status:

rs0:PRIMARY> rs.status()

{

"set" : "rs0",

"date" : ISODate("2016-08-26T09:15:09Z"),

"myState" : 1,

"members" : [

{

"_id" : 0,

"name" : "x-mongodb-01:27017",

"health" : 1,

"state" : 1,

"stateStr" : "PRIMARY",

"uptime" : 60029,

"optime" : Timestamp(1472202909, 3),

"optimeDate" : ISODate("2016-08-26T09:15:09Z"),

"self" : true

{

"_id" : 1,

"name" : "a.b.c.d:27017",

"health" : 1,

"state" : 0,

"stateStr" : "STARTUP",

"uptime" : 58860,

"optime" : Timestamp(0, 0),

"optimeDate" : ISODate("1970-01-01T00:00:00Z"),

"lastHeartbeat" : ISODate("2016-08-26T09:15:08Z"),

"lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"),

"pingMs" : 4

}

"ok" : 1

}

Where a.b.c.d is actually an IP. As you can see, the only secondary is stuck. What we have done so far:

Confirmed that x-mongodb-01 machine can use the mongo shell to reach a.b.c.d on port 27017

Confirmed that a.b.c.d can use the mongo shell to reach x-mongdb-01 on port 27017

Rebooted the production x-mongodb-01 machine with --replSet rs0

Started a.b.c.d mongo with --replSet 01

Issued rs.initiate() on x-mongodb-01

Issued rs.add({_id: 1, host: "a.b.c.d:27017", priority:0, hidden:true}) on x-mongodb-01

Did we miss something? The primary is mongo 2.4.12 and the secondary is 3.0.12.

James Green

unread,

Aug 26, 2016, 7:59:34 AM8/26/16

to mongodb-user

Solved.

The replica IP address was a public IP which a firewall forwards into the node hosting the instance. What is not clear in the documentation is that this secondary will not recognise itself and thus connect to the replica - which of course our firewall was preventing.

By renaming the host from a.b.c.d to the hostname of the replica and hard-coding the hostname of the replica into the /etc/hosts file of the primary, it all burst to life.