How can we skip snapshot or rows while using Postgres debezium connector.

[Razi Ahmed]

Hi Team,

Our use case is we have a table which have over millions of rows.We want to stream just latest rows only.Old data is of no use.For that purpose 

• Try to use snapshot.mode = never but it is not creating topic in kafka.
• If we run it on snapshot.mode=initial_only we have to parse all redundant rows that are not actually needed.

Need suggestion can we skip rows if we run with snapshot.mode=initial_only or is there should be any better approach.

[Chris Cranford]

Hi Razi -

When using the "snapshot.mode=never", topics for captured tables are created on-the-fly only when we observe a change event for said table. 

If you want Debezium to create the topics up-front, you could provide "snapshot.select.statement.overrides" configurations for each table where you give a custom select statement to perhaps only select a single row per table.  This reduces the volume of redundant data drastically but it also makes sure that the topics are created during the snapshot if that's something you need.  Otherwise you could manually create the topics up-front as well if the snapshot select overrides approach isn't desired.

HTH,
Chris

--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/f6b0960b-00dc-418a-8b57-457d31aa0de4n%40googlegroups.com.