Removing a MongoDB instance from a Replica Set when it's not the master

[shlomoid]

Hi,

I've been playing around with replica sets locally, and I managed to misconfigure the set. 

Basically, some mongos are configured using localhost and some using the hostname of the machine. This causes split brain in the set, and I even got some assertions popping up :)

Now, I'd like to reconfigure the set correctly - but the system won't let me do it easily : since there is a constant split brain going on, there is no elected master in place. I can't run rs.reconfig on the slave, since it's not the master (error: { "$err" : "not master", "code" : 10107 }).

What is the correct way to act here? What would I do if this happened in production? I couldn't find a way in the documentation that would remove the replication completely so I could reconfigure it - or just reconfigure it on the slave when there is no master in place.

Thanks!

Shlomo

[Kyle Banker]

If no master has been elected, then there's no way to reconfigure the
set. In this case, the only thing to do is to start over. This means
deleting the local.* files in each member's data directory.

> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to
> mongodb-user...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/mongodb-user?hl=en.
>

[shlomoid]

Thank you!

After digging through the documentation again, I've managed to find a page talking about this specifically:

http://www.mongodb.org/display/DOCS/Reconfiguring+a+replica+set+when+members+are+down

Indeed, removing local.* is the only option, as the documentation claims.

A theoretical question - what happens when we're storing some of our data in the 'local' database so that it doesn't get replicated? ( as the documentation http://www.mongodb.org/display/DOCS/About+the+local+database suggests) 

[Kyle Banker]

You shouldn't store application-level data in the local collection.
Will fix those doc now.

[Gaetan Voyer-Perrault]

> Basically, some mongos are configured using localhost and some using the hostname of the machine...

> What is the correct way to act here? What would I do if this happened in production?

Don't let this happen in production? :)

Seriously though, avoid using "localhost" for anything when running in production. Use DNS names or IP addresses on everything. When you run a sharded & replicated config, you end up with lots of Mongo processes

> A theoretical question - what happens when we're storing some of our data in the 'local' database so that it doesn't get replicated?

Same deal, avoid it if possible. It may work, but this has "hack" written all over it. 

If you want data that doesn't replicate, I would suggest running another instance of MongoD on a different port and simply not replicating it.

- Gates

[shlomoid]

:) Obviously, no one uses localhost for production - my main worry here was the little control I felt had over the replication process once it got going - and the inability to change it in certain scenarios. In MySQL there are ways to analyze a broken replication, and repair it - and I was trying to find the same thing in MongoDB.

And yes, using local has 'hack' written on it, I agree :)

[Kristina Chodorow]

You could always dump the local DB collections and restore them elsewhere with different names.

There is a way to manually reconfigure without deleting local (restart a slave w/out --replSet, do whatever you want to the config doc, increment the version number, and restart with --replSet), but you can totally mess up your config so you should generally avoid that.  This has come up before, so I made a bug to make this easier to reconfigure: https://jira.mongodb.org/browse/SERVER-2772.  A lot of people seem get wedged in a situation with all slaves when they're figuring out how replica sets work.

Remember that you can always start up a member of a set as a normal, non-replica set server by dropping the --replSet option (and changing the port # so the rest of the set doesn't recognize it).

On Wed, Mar 16, 2011 at 2:09 PM, shlomoid <shlo...@gmail.com> wrote:

:) Obviously, no one uses localhost for production - my main worry here was the little control I felt had over the replication process once it got going - and the inability to change it in certain scenarios. In MySQL there are ways to analyze a broken replication, and repair it - and I was trying to find the same thing in MongoDB.

And yes, using local has 'hack' written on it, I agree :)

--

[shlomoid]

On Wednesday, March 16, 2011 8:18:02 PM UTC+2, k wrote:

You could always dump the local DB collections and restore them elsewhere with different names.

Ah, so I can still dump data normally, even when the slave is unavaliable... I'm in the beginning processes of learning here, and I thought the slave won't let me read data when there is no master elected (I got an error message when I tried a find() ). Do I need to run the setSlaveOk to be able to dump slave data? (Which btw, is a terrible name :) )

 

There is a way to manually reconfigure without deleting local (restart a slave w/out --replSet, do whatever you want to the config doc, increment the version number, and restart with --replSet), but you can totally mess up your config so you should generally avoid that.  This has come up before, so I made a bug to make this easier to reconfigure: https://jira.mongodb.org/browse/SERVER-2772.  A lot of people seem get wedged in a situation with all slaves when they're figuring out how replica sets work.

Thanks for the bug report! That was swift :)

 

Remember that you can always start up a member of a set as a normal, non-replica set server by dropping the --replSet option (and changing the port # so the rest of the set doesn't recognize it).

So if I understand it correctly, a mongod instance would just ignore the "local" database when it's used without --replSet? 

[Kristina Chodorow]

mongodump should work on a slave.  You don't even have to have the server running to dump data from it, you can do:

$ mongodump -d local -c myLocalAppData --dbpath /path/to/your/db

and then you'd have myLocalAppData.bson, which you could load anywhere. 

You do have to set slaveOk (agree with you about the name) to do an ordinary find() on a slave. 

Thanks for the bug report! That was swift :)

Well, it's been percolating for a while :)

So if I understand it correctly, a mongod instance would just ignore the "local" database when it's used without --replSet?

It isn't really "ignored," it's just a normal database.  You can still use it, it just isn't used by MongoDB for replication.