2 member replica set - want to chante behavoir of: if secondary member is down, then the remaining member goes into seondary mode

[Brent Gracey]

I'm running a 2 member replica set - at the moment if the member which should run as secondary is down, then the remaining member goes into seondary mode - resulting in the whole system being down. 

What is the best way to update my config so that the db1 will vote itself as Primary even if it can't find db2?

The logs suggest adding an arbiter or giving db1 an extra vote. http://docs.mongodb.org/manual/reference/replica-configuration/#replica-set-reconfiguration-usage states "it is highly recommended each member has 1 or 0 votes" so is the quickest update to give db2 0 votes?

{

"_id" : "qsdb",

"version" : 1,

"members" : [

{

"_id" : 0,

"host" : "db1:27017",

"priority" : 2

},

{

"_id" : 1,

"host" : "db2:27017"

}

]

}

fg=rs.conf()

{

       "_id" : "qsdb",

       "version" : 1,

       "members" : [

               {

                       "_id" : 0,

                       "host" : "db1:27017",

                       "priority" : 2

               },

               {

                       "_id" : 1,

                       "host" : "db2:27017"

               }

       ]

}

Logs from db1:

BOOST_LIB_VERSION=1_49

Mon Oct 15 13:04:44 [initandlisten] options: { config: "/etc/mongodb.conf", dbpath: "/var/lib/mongodb", logappend: "true", logpath: "/var/log/mongodb/mongodb.log", replSet: "qsdb", rest: "true" }

Mon Oct 15 13:04:45 [initandlisten] journal dir=/var/lib/mongodb/journal

Mon Oct 15 13:04:45 [initandlisten] recover : no journal files present, no recovery needed

Mon Oct 15 13:04:50 [initandlisten] waiting for connections on port 27017

Mon Oct 15 13:04:50 [websvr] admin web console waiting for connections on port 28017

Mon Oct 15 13:04:51 [initandlisten] connection accepted from 127.0.0.1:43449 #1 (1 connection now open)

Mon Oct 15 13:04:54 [rsStart] replSet I am db1:27017

Mon Oct 15 13:04:57 [rsStart] replSet STARTUP2

Mon Oct 15 13:04:57 [rsMgr] replSet total number of votes is even - add arbiter or give one member an extra vote

Mon Oct 15 13:04:58 [rsSync] replSet SECONDARY

Mon Oct 15 13:05:00 [rsHealthPoll] couldn't connect to db2:27017: couldn't connect to server db2:27017

[Brent Gracey]

I updated the db2 to have 0 votes - and this has the desired effect of db1 staying in primary mode whether db2 is or is not present.

Any comments on the approach most welcome.

brent

[Shaun]

Hi Brent,

Here's my interpretation of what your goal is here.

1. Have two data nodes, with one preferred as primary
2. If the secondary goes down, keep the preferred primary
3. If the primary goes down, fail over to the secondary

In your example, you won't get the failover to the secondary because your secondary doesn't have any votes.  In general, changing the number of votes that each member has can lead to unexpected and hard to reason about behavior.

The standard way to do this is to change the priority of the nodes, rather than the number of votes.  http://docs.mongodb.org/manual/core/replication/#replica-set-node-priority  Because changing the number of votes is discouraged, this also means that you should always have an odd number of members in your replica set.  An arbiter in a separate datacenter (or small cloud instance) is one light weight way to do this.

One way to see why this is necessary for failover is to consider a situation with two nodes in separate datacenters.  If the network connection is lost but both nodes are up, both nodes will assume the role of primary which could lead to inconsistencies once they reconnect.

Hope this helps!  Let me know if you have any more questions about this.

Thanks!
~Shaun Verch

[Brent Gracey]

Hi Shaun,

your 3 points describes exactly what we are aiming for.

We have increased the priority of node 1 (see config) but this only gives us point 1, if either of the nodes fail, manual intervention is required for 2 or 3. By changing the number of votes, I now have 1 and 2, but still need manual intervention for 3. I think introducing an arbiter is the next step which could give the holy grail.

On Monday, October 15, 2012 4:34:00 PM UTC+1, Brent Gracey wrote:

[Shaun]

Hi Brent,

You are on the right track.  You need an odd number of nodes to ensure that you have a majority of votes no matter which one goes down.  An arbiter is a very light weight way to do this, and you can run it on a cheap (and in some cases free) cloud instance, since it doesn't actually do anything besides vote.