can't take down secondary in replica set

[Luke Ehresman]

I have two servers in a replica set.  rep2-1 and rep2-2.  I want to rebuild the secondary off of the primary (to compact the data files due to a high degree of fragmentation).  In the past, to do this I simply shut down the secondary, blow away the data directory, and start it back up and wait for it to sync  with the primary.

But when I tried to do that this morning, I'm getting a weird error.  I double checked, and rep2-1 is primary and rep2-2 is secondary. 

(on rep2-1)

> db.isMaster()

{

"setName" : "set2",

"ismaster" : true,

"secondary" : false,

"hosts" : [

"rep2-1:27018",

"rep2-2:27018"

],

"ok" : 1

}

(on rep2-2)

> db.isMaster()

{

"setName" : "set2",

"ismaster" : false,

"secondary" : true,

"hosts" : [

"rep2-2:27018",

"rep2-1:27018"

],

"primary" : "rep2-1:27018",

"ok" : 1

}

When I shut down rep2-2, any operations I try to make through mongos are giving me this error:

> db.my_collection.count()

Fri Nov  5 12:38:38 uncaught exception: count failed: { "errmsg" : "not master", "ok" : 0 }

> db.my_collection.find() 

error: { "$err" : "not master", "code" : 10107 }

I connected to rep2-1 and ran db.isMaster(), and now it says it's not master:

> use my_db

switched to db my_db

> db.isMaster()            

{

"setName" : "set2",

"ismaster" : false,

"secondary" : false,

"hosts" : [

"cuegg-data-rep2-1:27018",

"rep2-2:27018"

],

"ok" : 1

}

When I bring rep2-2 back up, everything is fine again.

Any suggestions?

Luke

--
Luke Ehresman
lu...@ehresman.org
Tebros Systems, LLC - http://tebros.com

[Markus Gattol]

Why are you using mongos if you only have a single rs with two nodes?
If you only have a simple replica set you don't need a mongos. Maybe
try not going trough a mongos but directly connect to the primary and
secondary using "mongo url:port/<database>"

[Luke Ehresman]

Our full setup is that we have two shards (shard1 and shard2) which are each a replica set (shard1=rep1-1,rep1-2; shard2=rep2-1,rep2-2)

It's the shard2 replica set that's giving me issues.

And by the way, I'm using 1.6.4 RC0

Luke

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com.
To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.

[Markus Gattol]

Sounds like some weird voting issue on this shard. Do you have an
arbiter? What are the priortiy values on the primary and secondary
(default is priority : 1).

[Kristina Chodorow]

If you only have two nodes, you can't take one down and still have a master. The system has 2 votes (on for each member), so a member needs 2 votes to become and stay master.  if you take down 2-2, 2-1 demotes itself because it only has 1 vote. 

If you want it to stay master when 2-2 goes down, consider adding an arbiter.

On Fri, Nov 5, 2010 at 9:54 AM, Luke Ehresman <lu...@ehresman.org> wrote:

I haven't changed the default priority, so it's 1 for all replicas.

No, no arbiter.

Why would a vote need to occur though?  I would think that the master would stay the master, regardless of the status of the secondary replicas.

Thanks for your help,

Luke

--

You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com.
To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.

--
Luke Ehresman
lu...@ehresman.org
Tebros Systems, LLC - http://tebros.com

[Luke Ehresman]

Thanks Kristina,  adding an arbiter solved the problem.

[Markus Gattol]

Luke, I compiled a FAQ item about when you would want an arbiter which
might be of interest:
http://www.markus-gattol.name/ws/mongodb.html#do_i_need_an_arbiter

[Luke Ehresman]

I haven't changed the default priority, so it's 1 for all replicas.

No, no arbiter.

Why would a vote need to occur though?  I would think that the master would stay the master, regardless of the status of the secondary replicas.

Thanks for your help,

Luke

--

You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com.
To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.