diff and sync two collections from different mongodb instances

[Dhi Aurrahman]

Hi,

Is there an efficient way to get delta of two collections from different mongodb instances, other than scanning it one-by-one? And after we get the delta we want to sync these two collections?

A use case would be: I have an app in remote area which has limited connectivity, hence I just can store the data "locally". Let say, we have `users` collection. At midnight, or at some point when we could manage a decent connection, I need to sync the "local" users collection with the "main" users collection stored in the main app.

Thank you!

Dhi

[Will Berkeley]

What is the "delta" of two collections? What does it mean to "sync" them? There's lots of valid ways to define those terms depending on the use case.

One approach for a situation like your local-remote example would be to mimic replication. Have the local instance be a standalone replica set so that it writes to an oplog. For the sync, your remote database can read from the oplog and apply the operations to sync the state. If the remote collection does not have any changes to it besides the sync and there's only one local instance syncing, then it's straightforward as there's no conflict between states to resolve. You just need to learn how to translate the oplog messages to the right CRUD operations.

With multiple local collections and a remote collection that may have its own local writes, things get much more complicated and there's no naive way to "delta" and "sync" multiple collections that you'd like to think of as syncing to one true data state. One collection might be in the process of being synced two or more other collections simultaneously, or be out of touch for a really long time, or be privileged as "the final word" for some subset of documents, etc, and there may be tricky conflict situations like updates from one local copy to a document deleted on another local copy.

-Will