Urgent : Not able to select a Primary

[Abhishek kumar]

Can someone explain me that the replica is not able to select a primary. What should I do ?

> rs.status()

{

"set" : "history_replica_set",

"date" : ISODate("2012-09-06T02:06:55Z"),

"myState" : 2,

"members" : [

{

"_id" : 3,

"name" : "mongo-a-3.persistence.us-east-1c.aws.talk.to:27017",

"health" : 1,

"state" : 2,

"stateStr" : "SECONDARY",

"uptime" : 367,

"optime" : {

"t" : 1346895661000,

"i" : 4

},

"optimeDate" : ISODate("2012-09-06T01:41:01Z"),

"lastHeartbeat" : ISODate("2012-09-06T02:06:54Z"),

"pingMs" : 0

},

{

"_id" : 4,

"name" : "mongo-a-1.persistence.us-east-1c.aws.talk.to:27017",

"health" : 1,

"state" : 2,

"stateStr" : "SECONDARY",

"optime" : {

"t" : 1346895661000,

"i" : 4

},

"optimeDate" : ISODate("2012-09-06T01:41:01Z"),

"self" : true

},

{

"_id" : 5,

"name" : "mongo-a-2.persistence.us-east-1d.aws.talk.to:27017",

"health" : 1,

"state" : 2,

"stateStr" : "SECONDARY",

"uptime" : 267,

"optime" : {

"t" : 1346895661000,

"i" : 4

},

"optimeDate" : ISODate("2012-09-06T01:41:01Z"),

"lastHeartbeat" : ISODate("2012-09-06T02:06:54Z"),

"pingMs" : 1

},

{

"_id" : 7,

"name" : "mongo-a-2.persistence.us-east-1b.aws.talk.to:27017",

"health" : 1,

"state" : 2,

"stateStr" : "SECONDARY",

"uptime" : 121,

"optime" : {

"t" : 1346895661000,

"i" : 4

},

"optimeDate" : ISODate("2012-09-06T01:41:01Z"),

"lastHeartbeat" : ISODate("2012-09-06T02:06:54Z"),

"pingMs" : 0

},

{

"_id" : 8,

"name" : "mongo-a-2.persistence.us-east-1c.aws.talk.to:27017",

"health" : 1,

"state" : 4,

"stateStr" : "FATAL",

"uptime" : 507,

"optime" : {

"t" : 1346860985000,

"i" : 3

},

"optimeDate" : ISODate("2012-09-05T16:03:05Z"),

"lastHeartbeat" : ISODate("2012-09-06T02:06:54Z"),

"pingMs" : 0

}

],

"ok" : 1

}

[Scott Hernandez]

What version are these nodes? Can you please post your rs.conf()?

I would suggest shutting down the fatal node if you have not yet.

> --
> You received this message because you are subscribed to the Google
> Groups "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com
> To unsubscribe from this group, send email to
> mongodb-user...@googlegroups.com
> See also the IRC channel -- freenode.net#mongodb

[Abhishek kumar]

2.0.5

{

"_id" : "history_replica_set",

"version" : 28,

"members" : [

{

"_id" : 3,

"host" : "mongo-a-3.persistence.us-east-1c.aws.talk.to:27017",

"priority" : 0

},

{

"_id" : 4,

"host" : "mongo-a-1.persistence.us-east-1c.aws.talk.to:27017"

},

{

"_id" : 5,

"host" : "mongo-a-2.persistence.us-east-1d.aws.talk.to:27017"

},

{

"_id" : 7,

"host" : "mongo-a-2.persistence.us-east-1b.aws.talk.to:27017"

},

{

"_id" : 8,

"host" : "mongo-a-2.persistence.us-east-1c.aws.talk.to:27017"

}

]

[Scott Hernandez]

Are these hosts in MMS, and can you check the rs.status() on all
nodes? Also, please post the logs from one of the secondaries in
gist/pastebin/pastie or some other service.

On Wed, Sep 5, 2012 at 10:22 PM, Abhishek kumar

[Abhishek kumar]

rs Status : http://pastebin.com/NScZ2n6n

Log : http://pastebin.com/A6PYUQSS

I fixed the issue. When I shutdown the mongo-a-2.persistence.us-east-1c.aws.talk.to node (whose state was FATAL) who was vetoing as shown from the logs. 

Please explain me the reason.

[Scott Hernandez]

Can you please create a server issue and post the logs from the fatal
node and one of the other nodes? http://jira.mongodb.org/browse/server

I am glad we were able to get you back up and running, and we will see
what we can find in the logs.

From a quick look at the rs.status() it seems like the fatal node
(which used to be primary) was ahead (with writes in the oplog) of the
other nodes and they didn't think they could elect each other (cause
it would veto having writes the others didn't), but I haven't looked
at everything yet, so we should get a better idea once you create the
server issue and someone can look at the logs.

Also, probably best to upgrade when you have a chance as there are
many fixes since 2.0.5.

On Wed, Sep 5, 2012 at 10:48 PM, Abhishek kumar

[Abhishek kumar]

will investigate at my end and then create the issue