replSet functioning

[Pratik Daga]

Hi,

I am trying to see how replSet works.

I figured out that if there are 2 nodes i.e one master/primary and one slave/secondry and if master goes down, slave don't vote itself to become master/Primary. It remain secondary.

But when there are 3 nodes, 1 master and 2 slave and if master goes down, one slave get selected as master by voting.

So,
is there any special case for 2 nodes?
or
Am I doing something wrong?
or
Is this is a bug?

I used this link as a reference
http://www.mongodb.org/display/DOCS/Replica+Set+Tutorial#ReplicaSetTutorial-FailingOver

can't see any special note for 2 nodes?
Also, as we discussed in meeting, on my mini mac every example is working!! so it should be something with OS.

[Kristina Chodorow]

A member needs a majority of votes to become primary. If you have 2 members, 2 is the only majority you can get. 1 isn't a majority, so if one member goes down, the other member can't become primary.  That's why the replica set tutorial you found has 3 members: 2 out of 3 is a majority, so the remaining members can elect a new primary.

[Pratik Daga]

thanks,

so if there are 2 nodes and master goes down how thing works? I mean does node which is up handles write and read independent of master and slave?

----- Original Message -----
From: Kristina Chodorow <kris...@10gen.com>
Date: Sunday, October 24, 2010 7:04 pm
Subject: Re: replSet functioning
To: mongo-node...@googlegroups.com

> A member needs a majority of votes to become primary. If you have 2 members,
> 2 is the only majority you can get. 1 isn't a majority, so if one member
> goes down, the other member can't become primary. That's why the replica
> set tutorial you found has 3 members: 2 out of 3 is a majority, so the

> remaining members *can* elect a new primary.

[Kristina Chodorow]

The node which is up can only handle reads.  It will be a slave until the other node is brought up again.  Writes will not work unless they are sent to a master.

[Pratik Daga]

Hi,

I have one doubt,

I am making replSetStatus.js which will ensure master,

I am making constructor replSetStatus(new Db("testdb", List<Serveraddress> list, otheroptions..), List<Serveraddress> list);

is Db type parameter fine?I mean I hope its not a bad coding decision.

I am using it because I have to run command DbCommand.createIsMasterCommand(Db);
for every server in list to check for master. And it takes Db as a parameter.

If you have to go through the code to answer it, I will figure it out, no worries.

[Kristina Chodorow]

It doesn't need to take db, you should just pass in the "admin" database.  The result of the command will be the same no matter which db is used.

[Pratik Daga]

never thought of that!! thanks a ton!!

[Kristina Chodorow]

np :)

[Pratik Daga]

So, I implemented a code in connection.js.

something like this

ReplSetStatus(List<Server>); // this is constructor
ensureMaster(); it returns latest master from the list (or NULL) every time you call it.

so you have to do something like this

replstat = new ReplSetStatus(List<ServerAddress>);
var latestMaster = replstat.ensureMaster();
and follow on.....................
like new Db("Mongo-DB", latestMaster, ....)

this is what you are looking for? or I am interpreting it wrong.
My code make sure latest master from the list when you call ensureMaster();

[Kristina Chodorow]

Cool, but that's not quite it... the class should work the same way the Connection class works (probably extend the Connection class) except it will find and use the replica set's master for all operations.  So, probably:

var conn = new ReplSetConnection(List<Server>); // automatically finds master (and other servers in the set)
new Db("Mongo-DB", conn);

The user shouldn't have to manually find the master and pass it along to MongoDB, because the master could change and the ReplSetConnection should be able to figure that out and start using the new master.

In the tests and examples, you should be able to replace any instance of Connection with ReplSetConnection and have it still work.

[Pratik Daga]

ok, var conn = ReplSetConnection(List<Server>);

and say conn have node1 which is master;

and then new Db("..", conn, ..)

and assume after 10 min node1 gets down!! user again have to call ReplSetConnection() to ensure new master,

so does user will put ReplSetConn() function in forever loop?

thanks,

[Kristina Chodorow]

If node1 goes down after 10 minutes, the next time the user does a query or a write, the operation will fail.  The Connection class already automatically tries to reconnect if the connection goes away, so you need to overwrite this functionality to try to figure out who the new master is from the list of possible hosts.

[Pratik Daga]

So.. I went through code of net.Connection which try to connect to the given port and host, independent of master connection.

However when I saw Db.js. I see db.open() try to set master connection.

db.executecommand() always attempt to execute command on masterconnection set by open().
even if that masterconnection is down.

so assume scenario.

repl = new ReplSetConnection(List<ServerAddress>)

db = new Db("....", repl....);
db.open....

db.executecommand(some db_command) // this will try to use masterconnection for execution.
....
....
... //assume now masterconnection goes down
now ReplSetConnection should set masterconnection to new master from list.

so that if we again execute code db.executecommand(some db_command); it wont fail

is that what we need? if yes!! don't you think we just need to add up code in db.js to ensuremaster everytime we execute command instead of adding things into autoreconnect functionality of connection.js?

when I see in connection.js I can't see anything promising(probably my lack of understanding) which can ensure master. However in net.createConnection I see connect and doConnect calling each other. so do you want me to do something in net.js to ensuremaster. ??

I might be missing something, but I am confused here.
I am sorry for keep bugging you. :(

[Kristina Chodorow]

is that what we need?

Yes!!
 

if yes!! don't you think we just need to add up code in db.js to ensuremaster everytime we execute command

We can't do this, unfortunately, because calling ensuremaster is expensive.  So, you have to wait for something to fail, then find a new master.

 instead of adding things into autoreconnect functionality of connection.js?

I haven't looked at the code much, you probably know it better than I at this point.  I suggested connection.js because that's where all of the other drivers do it, but the nodejs driver seems a bit differently organized.  Put it in whichever file makes sense.

I am sorry for keep bugging you. :(

No problem at all!  This is tricky stuff and important to get right.  Keep the questions coming!

[Pratik Daga]

>
> We can't do this, unfortunately, because calling ensuremaster is expensive.
> So, you have to wait for something to fail, then find a new master.

I agree that ensuremaster is expensive but don't you think keeping it in loop forever(to see for fail) is much more costlier than calling it only before executing command ?

[Kristina Chodorow]

No, don't loop forever anywhere!

If excutecommand fails, call ensuremaster. 

Say you're using a "normal" mongod (just one process, connect to it with new Connection()). If you're doing operations and suddenly kill the database in the middle of them, the nodejs driver should throw an exception or something.  What you want to do is, before that exception is thrown, if the connection is a replica set connection, figure out if there's a new master you can use instead.

[Pratik Daga]

ok, i think I am getting your point!!

Lets see, hoping to commit this stuff before next meet!!

Thanks.

[Kristina Chodorow]

Cool!

[Pratik Daga]

so,

I made up a code which take care of fail over.

my Logic

if(get diconnected) {

call admin db and do ismaster command and
get master/secondry in server list.

}

to call admin db I did something like this within connection.js
new Db("admin", new ReplicaSet<list>, ..)
Db.execute_is_mastercomand for each server in list
and update master connection

I am not sure is that right thing to do?
Is that fine or you think there is better way to use admin db to execute command.

However I am sure that code will take care of fail over :).

[Kristina Chodorow]

Sounds good, let's talk about it tonight.