Debezium Embedded Engine.

[Joseph Aliase]

I'm using Debezium Embedded Engine (version 1.9.5(old versionl) and need to confirm the offset flushing behavior.

My Question:

When offset.flush.interval.ms triggers a flush, does Debezium call OffsetBackingStore.set() with:

Only the offsets for records explicitly marked via RecordCommitter.markProcessed(), OR
All offsets for records received, regardless of whether markProcessed() was called?
My Understanding:

I believe it's #1 - only offsets for records marked via markProcessed() are flushed. This ensures at-least-once semantics: if I skip calling markProcessed() due to a processing failure, that offset won't be committed and the record will be reprocessed after restart.

Context:

In my implementation, I only call markProcessed() after successful downstream processing. If processing fails, I throw an exception without calling markProcessed(). I want to confirm the uncommitted offset won't get flushed during the next offset.flush.interval.ms cycle.

Can someone confirm this behavior?

Thanks!

[Joseph Aliase]

Chris Cranford - Could you please take a look at my request? 

[Chris Cranford]

Hi,

You are correct. When you call "markProcessed()", Debezium enqueues that offset information to be flushed on the next offset flush interval.

HTH,
-cc

--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/debezium/6cdcbde4-2ee4-4699-ac1a-4af50cab1b6cn%40googlegroups.com.

[Joseph Aliase]

Thank you Chris. 

To view this discussion visit https://groups.google.com/d/msgid/debezium/171193e3-e364-4789-a1bb-f82090648883%40gmail.com.