Mongo connector lag

[Ankur Mishra]

Hi,

We are using debezium mongo connector enabled cdc pipeline in place.

We were seeing lag on connector which used to go up till oplog window which was 5 hrs.

We thought of increasing oplog window and now lag goes upto 10 hrs (which is oplog window now). We have tried increasing batch.size and reducing poll time but that also didn't help. Also cpu usage is 40% for debezium ecs container which shows that cpu is not the blocker. It seems we are missing some config either on debezium or some mongo config. Once oplog window reaches to max (10 hrs), lag starts coming down.  Can anyone help with this.

Regards,

Ankur

[jiri.p...@gmail.com]

Hi,

what Debezium version are you on?  Could you share https://debezium.io/documentation/reference/2.4/connectors/mongodb.html#connectors-snaps-metric-queueremainingcapacity_mongodb metric?

Could you share the configuration?

J.