Mixing mongodb versions in a replica set

[Dave Brueck]

Hi, we are running a mongo 2.6.8, 5 member replica set on Ubuntu Linux 14.04 and are looking to upgrade to 3.2.x and also switch the storage engine to Wired Tiger in the process. We do not use sharding. The tricky part for us is that we operate a service and can't have scheduled maintenance windows of very big size, so our hope is to minimize downtime to a few 10s of seconds early some morning. Our hope is that we could upgrade the cluster by just adding a new node to the cluster - one that is running newer mongo and configured to use WT storage - let it get all sync'd up, and then remove one of the nodes running 2.6, and repeating the process until all the nodes are up to date.

So rather than actually upgrading nodes, we'd really be doing a fresh install on each one - we don't have to run a node's copy of the data through any sort of upgrade process but instead will treat it like temporarily expanding the replica set's size with a new server. Our desire to do it this way is to minimize downtime but also to allow us to proceed slowly - we want to bring up a new node and just let the dust settle for awhile, run some tests, etc. before proceeding. There are different elements of risk no matter which way we go, but our preference is for a process that changes few things at a time even if the overall end-to-end time is longer (we like lots of incremental changes spread out over time).

Some questions:

0) Based on the docs (https://docs.mongodb.org/manual/release-notes/3.2-upgrade/#upgrade-replica-set) it sounds like /no/ members of the replica set can be running 3.2 until they are /all/ running at least 3.0 - is that correct? If so, then we should plan on doing our node-by-node upgrade to 3.0 and then once that's all done, repeating the process to get to 3.2. I'm asking because obviously jumping from 2.6 to 3.2 would be nice, but understand that that's a pretty huge leap.

1) Is this really a viable plan, to have a replica set with both 2.x and 3.x nodes simultaneously? It's perfectly ok with us if the 3.x nodes can't use all the newer functionality til the migration is done - we're just looking for a process that gets us there with limited downtime. If so, is there any special way we need to configure the 3.x nodes while they coexist with the 2.x nodes to make sure they all play nicely together? Similarly, once the migrations are done, are there any settings we should change to take advantage of newer functionality?

2) How can we go about minimizing the number of primary elections? (or other things that would cause the replica set to be temporarily unavailable) As in, could we add new nodes as secondaries and have them be used and then at some point force an election to one of the new primaries? (I know how to force an election to a new node, but am less sure about adding secondaries and getting traffic to them without forcing an election - e.g. doesn't a rs.reconfig always trigger an election?) Our application uses a read preference of 'nearest' for a lot of our traffic, as in many cases eventually consistent data is fine, so most of our secondaries handle a lot of traffic on an ongoing basis, so we'd want to be able to add the 3.x secondaries and get traffic to them to test things out before going too far. It'd just be nice to add them without causing an election each time.

Thanks in advance for any help or recommendations!

-Dave

[Wan Bachtiar]

0) Based on the docs (https://docs.mongodb.org/manual/release-notes/3.2-upgrade/#upgrade-replica-set) it sounds like /no/ members of the replica set can be running 3.2 until they are /all/ running at least 3.0 - is that correct?

Hi Dave,

That is correct. In order to upgrade to MongoDB v3.2, all of the replica set members must be running v3.0. You can not have a mix of v2.6 and v3.2 in your replica set.

1) Is this really a viable plan, to have a replica set with both 2.x and 3.x nodes simultaneously? It’s perfectly ok with us if the 3.x nodes can’t use all the newer functionality til the migration is done - we’re just looking for a process that gets us there with limited downtime. If so, is there any special way we need to configure the 3.x nodes while they coexist with the 2.x nodes to make sure they all play nicely together?

Just to clarify that stating v2.x and v3.x is very broad. I assumed that you are meaning to say v2.6 and v3.0.
See MongoDB Version Numbers for more info on MongoDB release series.

As per the upgrade notes, before enabling the new WiredTiger storage engine, ensure that all replica set members are running at least v2.6.8, and preferably v3.0.8 or newer.

You can perform a rolling upgrade of a replica set with MongoDB v2.6 and v3.0. Although running a mixed version is supported for the purposes of upgrading your replica set, you should not run with this configuration for longer than is required.

Similarly, once the migrations are done, are there any settings we should change to take advantage of newer functionality?

There are a number of updates in v3.0. If you are wanting to take advantage of the new WiredTiger storage engine, see change storage engine to WiredTiger for the steps involved.

2) How can we go about minimizing the number of primary elections? (or other things that would cause the replica set to be temporarily unavailable) As in, could we add new nodes as secondaries and have them be used and then at some point force an election to one of the new primaries? (I know how to force an election to a new node, but am less sure about adding secondaries and getting traffic to them without forcing an election - e.g. doesn’t a rs.reconfig always trigger an election?)

Few points to clarify:

• rs.reconfig() does not always trigger an election (at least in v2.6.8+).
• You can add a replica set member using rs.add().
• You can force a member to be primary using database commands without rs.reconfig(). An election would be performed.

Our application uses a read preference of ‘nearest’ for a lot of our traffic, as in many cases eventually consistent data is fine, so most of our secondaries handle a lot of traffic on an ongoing basis, so we’d want to be able to add the 3.x secondaries and get traffic to them to test things out before going too far. It’d just be nice to add them without causing an election each time.

As with any systems upgrade, you should always test in a development or staging environment first. You should test the client application code as well for any code/driver incompatibility issues before testing in production.

Also I would recommend to check the compatibility changes notes carefully as well. A few points to be aware of:

• Once upgraded to v3.0, you cannot downgrade to a version lower than v2.6.5.
• If you upgrade to v3.0 and have run authSchemaUpgrade, you cannot downgrade to v2.6 without disabling --auth or restoring a pre-upgrade backup. The authSchemaUpgrade discards MONGODB-CR authentication credentials used in v2.6.

Once you have all the members upgraded to v3.0, please check the upgrade notes for v3.0 to v3.2 which is different from v2.6 to v3.0.

Lastly make sure to use the latest stable release of MongoDB for bug fixes and general enhancements. Currently for v3.0 it is v3.0.8, and for v3.2 it is v3.2.1

Best regards,

Wan.

​

[Dave Brueck]

Thanks Wan, for a very thorough and helpful response - this is just the info I needed!

-Dave