Collection UUID is not preserved by mongorestore

[Srihari Prabhakar]

Hi,

In MongoDB 3.6, when restoring the database or any specific collection from dumps, mongorestore doesn't create the new collections with same UUIDs. UUIDs on restored DB is different from UUIDs on source DB. But, this UUID is used during replication. So, when I restore a stale secondary using mongodumps, it fails as collection UUIDs are different in primary and the secondary. Is there a way to fix this?

-- On primary

rep01:PRIMARY> db.testcollection.exists().info.uuid
UUID("416c4740-8181-4162-a1ed-79489421c6ca")

-- On restored secondary

rep01:SECONDARY> db.testcollection.exists().info.uuid
UUID("fdc71550-c817-49e3-abcc-5be8f37c2fc2")

-- Once new records are inserted on primary secondary fails due to error in replication

2018-03-17T05:06:21.324+0000 F REPL     [repl writer worker 7] writer worker caught exception: NamespaceNotFound: Failed to apply operation due to missing collection (416c4740-8181-4162-a1ed-79489421c6ca): { ts: Timestamp(1521263181, 1), t: 1, h: 2646940537054691738, v: 2, op: "i", ns: "test.testcollection", ui: UUID("416c4740-8181-4162-a1ed-79489421c6ca"), wall: new Date(1521263181321), o: { _id: ObjectId('5aaca24dddcb649ae3623fbd'), n: 1004.0 } } on: { ts: Timestamp(1521263181, 1), t: 1, h: 2646940537054691738, v: 2, op: "i", ns: "test.testcollection", ui: UUID("416c4740-8181-4162-a1ed-79489421c6ca"), wall: new Date(1521263181321), o: { _id: ObjectId('5aaca24dddcb649ae3623fbd'), n: 1004.0 } }
2018-03-17T05:06:21.324+0000 F -        [repl writer worker 7] Fatal assertion 16359 NamespaceNotFound: Failed to apply operation due to missing collection (416c4740-8181-4162-a1ed-79489421c6ca): { ts: Timestamp(1521263181, 1), t: 1, h: 2646940537054691738, v: 2, op: "i", ns: "test.testcollection", ui: UUID("416c4740-8181-4162-a1ed-79489421c6ca"), wall: new Date(1521263181321), o: { _id: ObjectId('5aaca24dddcb649ae3623fbd'), n: 1004.0 } } at src/mongo/db/repl/sync_tail.cpp 1230 2018-03-17T05:06:21.325+0000 F -        [repl writer worker 7]


***aborting after fassert() failure

[Stephen Steneker]

On Saturday, 17 March 2018 16:12:07 UTC+11, Srihari Prabhakar wrote:

In MongoDB 3.6, when restoring the database or any specific collection from dumps, mongorestore doesn't create the new collections with same UUIDs. UUIDs on restored DB is different from UUIDs on source DB. But, this UUID is used during replication. So, when I restore a stale secondary using mongodumps, it fails as collection UUIDs are different in primary and the secondary. Is there a way to fix this?

Hi Srihari,

This is the expected outcome if you write data directly to a secondary. Replica set data integrity relies on all members having consistent oplog history and data provenance. Any changes you make on a replica set member in offline mode will not be written to the oplog and may cause the replica set member to diverge from the rest of the replica set. If a consistency violation is encountered, the replica set member will shutdown with a fatal assertion noting the issue (as you have encountered).

The collection UUIDs in MongoDB 3.6 help identify data provenance: dropping and recreating a collection will intentionally result in a new UUID which indicates that a namespace has been reused.

The only supported procedures for re-syncing a stale secondary are initial sync or a file copy restore. See: Resync a Member of a Replica Set in the MongoDB manual for more information.

If you want to restore individual collections or databases, you need to mongorestore via the current primary so the changes propagate using replication. mongorestore cannot be used to restore a stale secondary.

Regards,

Stennie