MongoDB Debezium connector loses committed offset on restarting or changing the config | resume_token

[Nikhil Jain]

Hi Team, 
I am using mongoDB debezium connector on kafka-connect to ingest data from MongoDB bin logs. 

When I tried to pause--> change config --> resume the connector, it loses it's offset. 

Also when I tried to directly change config for running connector (using change-config API )  debezium connector is losing it's source offset and start ingesting data from scratch from mongoDB bin-logs.

Configuration change is like changing batch_size and queue_size to monitor the throughput.

Queries : 

Is there any way so that source-offset (means how much data connector has read from bin-logs)  couldn't be lost and on restarting the connector, it starts ingesting data from the committed offset only not from scratch.

Also from where can we get how much data Debezium connector has read from source? (here mongoDB bin-logs)

Suspected issue: 

When I tried to look for committed offsets in 'connect-offsets' topic , for some collections I am receiving 'resume_token' field and for some collections it is missing. It seems 'resume_token' field is used to maintain the committed offset. And for collections, where it is not present, offset is being lost.

Any help would be appreciated 

Attachments showing resume_token:

Regards:
Nikhil Jain

[jiri.p...@gmail.com]

Hi,

is this visible with our tutorial too?

Thanks

J.

[Nikhil Jain]

Hi Jiri,
which tutorial are you talking about?


Regards
Nikhil Jain

On Tue, 27 Sept 2022 at 12:01, jiri.p...@gmail.com

> --
> You received this message because you are subscribed to a topic in the Google Groups "debezium" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/debezium/MqFPT9hr8kI/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to debezium+u...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/3b4d34e7-a65e-45ca-a233-882972bbe887n%40googlegroups.com.

[Vojtech Juranek]

hi,

> which tutorial are you talking about?

I believe it's this one:

https://github.com/debezium/debezium-examples/tree/main/tutorial#using-mongodb

Vojta

[Akshay Saini]

Hey Nikhil,

I think your debezium connector is running under snapshot mode. You will not fine resume_token until connector has been shifted from snapshot to change_steam mode.

As debezium doc states :
"when the connector sees a replica set for the first time, it looks at the oplog to get the last recorded transaction and then performs a snapshot of the primary’s databases and collections. When all the data is copied, the connector then creates a change stream from the position it read earlier from the oplog.
...
snapshot will continue until it has copied all collections that match the connector’s filters. If the connector is stopped before the tasks' snapshots are completed, upon restart the connector begins the snapshot again."

I hope it helps

Best Regards

Akshay Saini

[George H]

Hi there,

My name is George form a recruitment consultancy in the UK.

We are recruiting for a Debezium Consultant for a remote working contract.

If you are interested or know anyone who has excellent Debezium experience please mail geo...@digisourced.com and rebecca...@recann.co.uk