bad repl set config - assertion:13122

[Eytan Daniyalzade]

Hey, 

I have a cluster of 4 nodes, that has been running fine for months now. And, all of a sudden when I am trying to remove a node from the replicaset, the primary is giving me the following assertion. I am running mongo 1.8.1 on all the nodes on ubuntu 10.04.

(primary_mongo_shell) >>rs.remove('host01')

{

"assertion" : "bad repl set config?",

"assertionCode" : 13122,

"errmsg" : "db assertion failure",

"ok" : 0

}

(primary_mongo_shell) >>rs.conf()

{

"_id" : "mongo",

"version" : 5000,

"members" : [

{

"_id" : 18,

"host" : "host05:27017"

},

{

"_id" : 27,

"host" : "host04"

},

{

"_id" : 31,

"host" : "host06"

},

{

"_id" : 33,

"host" : "host01"

}

]

}

(primary_mongo_log):

Thu Aug 16 10:16:15 [conn1460978] query admin.$cmd ntoreturn:1 assertion  command: { replSetReconfig: { _id: "cellar", version: 5001, members: [ { host: "host05:27017", _id: 18 }, { host: "host04", _id: 27 }, { host: "host06", _id: 31 }, { host: "host01", _id: 33 } ] } } reslen:141 1557ms

also, host05 is the primary.

Any ideas? How can I get out of this situation?

Thanks,

--Eytan

[Eytan Daniyalzade]

Ah, here is the answer. 

https://github.com/mongodb/mongo/blob/r1.8.1/db/repl/rs_config.cpp. For some reason, the maximum replicaset version that is allowed is 5000 in version 1.8.1?? Here is the relevant code snippet:

        configAssert( version < 0 /*unspecified*/ || (version >= 1 && version <= 5000) );

Any advise in how we can get out of not being able to update the configuration? We would like to override the configuration with an earlier version, say '2'. But, the mongo code does not allow downgrading the replicaset version either. 

[Jenna]

Hello Eytan,
The limit on version number was removed in MongoDB 2.0. Is upgrading
a possibility? Here is the documentation for upgrading a replica set
to version 2.0 (just as a side note- MongoDB version 2.2 is due to be
released very soon): http://www.mongodb.org/display/DOCS/2.0+Release+Notes#2.0ReleaseNotes-Upgrading

If upgrading is not a possibility, you will have to take downtime in
order to reset the version numbers on each node in the replica set.
If you would like to continue using 1.8 and would like additional
information about this process, please let me know.

On Aug 16, 11:56 am, Eytan Daniyalzade <ey...@chartbeat.com> wrote:
> Ah, here is the answer.
>

> https://github.com/mongodb/mongo/blob/r1.8.1/db/repl/rs_config.cpp<https://github.com/mongodb/mongo/blame/r1.8.1/db/repl/rs_config.cpp>.