Outbox Pattern and SNAPSHOT

[Dylan Thomas]

Hi all,

We're currently using Debezium + CDC to replicate data from one Postgres DB to a second DB to create materialized views. Not a fan of the materialized view pattern but that's another topic...

The source DB stores most data as unstructured JSOB and our transformation layer (sink handler) became way too complex leading to bugs. We're considering switching to an Outbox Pattern to reduce transformation complexity and publish structured data based on AVRO schemas.

My question is, how can SNAPSHOT functionality be implemented when using the Outbox pattern? My guess is you can't. We currently rely on snapshotting to address operational issues and data corruption due to code bugs.

Is there an alternative or practices that provide similar capabilities?

[jiri.p...@gmail.com]

Hi,

please check https://debezium.io/documentation/reference/1.4/integrations/outbox.html#_runtime_configuration_options

That would allow outbox wokring even for snapshotting.

J.

[Dylan Thomas]

Thanks jiri.p...

We do not use Quarkus, or JAVA for that matter. Is that a requirement to make this work? Are the specific setting I should look at for snapshotting? 

[jiri.p...@gmail.com]

Hi,

I am not sure I undertnd the problem fully then. If you use the outbox pattern and if you keep the rows in the outbox table then the snapshot should/will be possible to pickup the messages.

J.

[Dylan Thomas]

Makes sense, I was confused because the settings page you linked looked specific to quarkus. Thanks!!