Oplog format

[Zt Zeng]

Is there any document list all possible format of Mongo's oplog format? In my last search, the following is the 'right' format for a update operation, in which has field `o.$set`:

{"ts":Timestamp(1521530692,1),"t":NumberLong("5"),"h":NumberLong("-384939294837368966"), "v":2,"op":"u","ns":"foo.bar","o2":{"_id":"L0KB$fjfLFra"},"o":{"$set":{"apns":"[]"}}}

But after some running of parsing oplog, we find a different log which doesn't has field `o.$set`, as this jira says, it is normal:

> believe you're seeing this because the $setoperator was not used to update the document; this is expected if the document was overwritten entirely as opposed to just a single field being changed.

 So, I am wonder if there any detailed format description of oplog? Thanks.

[Stephen Steneker]

On Tuesday, 2 October 2018 09:41:03 UTC+10, Zt Zeng wrote:

Is there any document list all possible format of Mongo's oplog format?

Hi,

The oplog format is used by MongoDB replication and subject to change between major releases of MongoDB. I'm not aware of any documentation on all possible variations outside of reviewing the code on Github.

However, MongoDB 3.6 introduced a Change Streams API which is a supported interface for third party integration and definitely preferable to using the oplog directly. MongoDB 4.0 adds some further improvements to Change Streams including database-wide or deployment-wide change streams and the option to specify a start time.

If Change Streams does not address your use case, can you provide more detail on what you are trying to achieve and the version(s) of MongoDB you need to support? Regards,  Stennie

[Zt Zeng]

Hi,

We are doing some data sync from mysql/mongo to ES for searching. Thanks a lot for your suggestions about change stream which is what we expected. However, we are using mongo 3.2 for the time being. We can migrate to change stream when mongo is updated, and now, we may be have to try to tail the oplog.

[Stephen Steneker]

On Thursday, 4 October 2018 08:09:45 UTC+10, Zt Zeng wrote:

We are doing some data sync from mysql/mongo to ES for searching. Thanks a lot for your suggestions about change stream which is what we expected. However, we are using mongo 3.2 for the time being. We can migrate to change stream when mongo is updated, and now, we may be have to try to tail the oplog.

Hi,

Since you are using an older version of MongoDB you might want to look into mongo-connector, a community tool which currently supports up to MongoDB 3.4.

Also note that MongoDB 3.2 reached end of life last month (Sept, 2018) so you should definitely be planning to upgraded to a newer and supported server version soon.

Regards,

Stennie

[Zt Zeng]

Hi,

Thanks for your suggestions. We might can't use mongo-connector because need to listen to change from both mysql/mongo, but we can learn from that tool to see how it handles oplog. Thanks anyway.