Microservice 'Change Data Capture' via @Outgoing Message 'Channel Adapter' and 'Outbox' pattern

[Gordon Hutchison]

I am (still) intrigued by what one does with state under microservices without distributed transactions.

An interesting approach is that of (forgive the buzzword bingo) 'change data capture' with the 'outbox' pattern

and channel adapters.

Background reading for area:

   https://en.wikipedia.org/wiki/Change_data_capture

   https://microservices.io/patterns/data/application-events.html

   https://debezium.io/blog/2019/02/19/reliable-microservices-data-exchange-with-the-outbox-pattern/

   https://www.confluent.io/blog/using-logs-to-build-a-solid-data-infrastructure-or-why-dual-writes-are-a-bad-idea/

   https://evandontje.com/2017/12/13/avoid-dual-writes-with-sql-server-and-change-tracking-part-1/

   (write through database para in) https://www.confluent.io/blog/messaging-single-source-truth/

Now it may be that this is useful stuff but a question is - Is there additional benefit to it being part of MicroProfile?

To this end, I was wondering what something like a Debezium Connector for MicroProfile Reactive Messaging @Outgoing

would help with? and what it would look like...

...and then my brain exploded. So I thought I'd better write the idea (not new, just new to me) down (here).

Gordon

[Gunnar Morling]

Hi Gordon,

Thanks for bringing this up (for those not aware, I'm the lead of Debezium, so naturally this thread caught my attention ;)!

> To this end, I was wondering what something like a Debezium Connector for MicroProfile Reactive Messaging @Outgoing

would help with? and what it would look like...

Could you elaborate what you have in mind here? Debezium itself works at the database level, so its usage is transparent to applications writing to the database. Also there's no direct connection from the application to Kafka or any other messaging broker, this all is done "under-the-hood" by tailing the DB logs. So I'm curious about the potential usage you envision.

> Is there additional benefit to it being part of MicroProfile?

One thing I could see being useful is standardized means to produce the events that should be written into the outbox table. In our current PoC [1] we have a bit of glue code [2] for that, which lets the application code do that using CDI events [3]. A standardized API for this could be useful potentially. I reckon that it probably would be out of the scope of MicroProfile how the entries in that table are then processed by a CDC engine such as Debezium.

--Gunnar

[1] https://github.com/debezium/debezium-examples/tree/master/outbox

[2] https://github.com/debezium/debezium-examples/tree/master/outbox/order-service/src/main/java/io/debezium/examples/outbox/order/outbox

[3] https://github.com/debezium/debezium-examples/blob/master/outbox/order-service/src/main/java/io/debezium/examples/outbox/order/service/OrderService.java#L32-L40

[Gordon Hutchison]

On Tuesday, 16 April 2019 07:52:49 UTC+1, Gunnar Morling wrote:

Hi Gordon,

Thanks for bringing this up (for those not aware, I'm the lead of Debezium, so naturally this thread caught my attention ;)!

> To this end, I was wondering what something like a Debezium Connector for MicroProfile Reactive Messaging @Outgoing

would help with? and what it would look like...

Could you elaborate what you have in mind here? Debezium itself works at the database level, so its usage is transparent to applications writing to the database. Also there's no direct connection from the application to Kafka or any other messaging broker, this all is done "under-the-hood" by tailing the DB logs. So I'm curious about the potential usage you envision.

Well my thoughts are just unfolding but for example when reading "You also provide the engine with a function that it will call for every data change event produced by the connector."

at https://debezium.io/docs/embedded/ this made me think of that data source being wired up as a Publisher of a MicroProfile Reactive Stream and how that could be wired to an MicroProfile Reactive Messaging @Outgoing that sent it off to a provided distribution mechanism.

I was also interested in using the shape/signature of MicroProfile Reactive Messaging annotated methods to indicate that they are, by some convention,

being the source or destination of CDC events for a particular domain type and thus create an abstraction that is slightly above the level of a particular database

table. 

By I have to admit that these are just possibilities swirling in my head.

I am trying to implement a 'no fear, share early share often' poilicy for ideas.

Gordon

[Gordon Hutchison]

Hi Gunnar,

Thanks a lot for taking the time to respond. It is interesting what you say.

Regarding:

>>> A standardized means to produce the events that should be written into the outbox table.

I will try to do some more background reading on this, there was some talk of a MicroProfile data access

API, possibly reactive, possibly on top of r2dbc from Andy Guibert and others in this group.

I wonder if feeding the outpox table automatically via that would be more portable than the current lower

level approaches?

Thanks again for engaging.

Gordon.

On Tuesday, 16 April 2019 07:52:49 UTC+1, Gunnar Morling wrote:

[Gunnar Morling]

Am Dienstag, 16. April 2019 10:38:33 UTC+2 schrieb Gordon Hutchison:

On Tuesday, 16 April 2019 07:52:49 UTC+1, Gunnar Morling wrote:

Hi Gordon,

Thanks for bringing this up (for those not aware, I'm the lead of Debezium, so naturally this thread caught my attention ;)!

> To this end, I was wondering what something like a Debezium Connector for MicroProfile Reactive Messaging @Outgoing

would help with? and what it would look like...

Could you elaborate what you have in mind here? Debezium itself works at the database level, so its usage is transparent to applications writing to the database. Also there's no direct connection from the application to Kafka or any other messaging broker, this all is done "under-the-hood" by tailing the DB logs. So I'm curious about the potential usage you envision.

Well my thoughts are just unfolding but for example when reading "You also provide the engine with a function that it will call for every data change event produced by the connector."

at https://debezium.io/docs/embedded/ this made me think of that data source being wired up as a Publisher of a MicroProfile Reactive Stream and how that could be wired to an MicroProfile Reactive Messaging @Outgoing that sent it off to a provided distribution mechanism.

Ah, yes, gotcha. We had been exploring this a bit some time ago: expose a stream of CDC events as ReactiveStreams using that embedded engine you mention. This experiment stalled a bit, mainly due to the need for maintaining some state in such publisher (it must regularly persist the offset in the DB log, so it can resume from that point after a restart. It would be very interesting to take this up again as it enables many interesting use cases also if there's no full-blown message broker available.

I was also interested in using the shape/signature of MicroProfile Reactive Messaging annotated methods to indicate that they are, by some convention,

being the source or destination of CDC events for a particular domain type and thus create an abstraction that is slightly above the level of a particular database

table. 

Some means of conversion/mapping would be needed for that probably. It's a bit blurry to me how this could look like. What definitely could be interesting is a way to mark methods as receivers for specific CDC event types (corresponding to tables), with parameter types already providing some kind of un-marshalling (e.g. JsonObject or even strongly typed DTOs). OTOH you probably could get this as of today by using the regular Kafka deployment style of Debezium and having @Incoming methods for specific topics (by default, the changes from one table are streamed to one topic).

By I have to admit that these are just possibilities swirling in my head.

I am trying to implement a 'no fear, share early share often' poilicy for ideas.

+1 That's very much appreciated.

I'm very glad to see this discussion being started and I'll be happy to contribute to it. We also could prototype different things in Debezium; any help with that will be welcomed of course :)

[Gunnar Morling]

Am Dienstag, 16. April 2019 10:49:29 UTC+2 schrieb Gordon Hutchison:

Hi Gunnar,

Thanks a lot for taking the time to respond. It is interesting what you say.

Regarding:

>>> A standardized means to produce the events that should be written into the outbox table.

I will try to do some more background reading on this, there was some talk of a MicroProfile data access

API, possibly reactive, possibly on top of r2dbc from Andy Guibert and others in this group.

I wonder if feeding the outpox table automatically via that would be more portable than the current lower

level approaches?

Two thoughts on that: 

The writes to the outbox table must happen in the same TX as the corresponding change to the application's actual business tables. The idea behind this is to avoid potential inconsistencies between the DB and the messaging broker (which in case of Kafka cannot share a TX with the DB), by only updating the DB and then update the messaging broker via CDC.

The events written to the outbox table form an explicit contract with downstream consumers, so the application typically needs to have full control over their structure; this allows to decouple the application's internal model from the event schema exposed to others via the messaging broker. So if one wanted to automate that, there'd have to be a way for customizing the outbox events so they can differ from the application's internal data schema. I felt doing the write to the DB and then emitting a CDI event (as shown in the linked PoC) is easy enough for that.

[clement escoffier]

Hello,

Looking at:

// Create the engine with this configuration ...

EmbeddedEngine engine = EmbeddedEngine.create()

        .using(config)

        .notifying(this::handleEvent)

        .build();

It seems to be already possible to get a Publisher using the `handleEvent`. Yes it would need a buffer to handle the back-pressure (so it’s far form perfect, but would work). 

[I’m actually wondering what happens if the handleEvent method blocks]. So doing a Reactive Messaging connector would be trivial, without any change in Debezium itself.

Also, related to Reactive Messaging and Debezium, there is a demo showing how a Reactive Messaging application ingesting the CDC produced by Debezium (via Kafka): https://debezium.io/blog/2019/03/14/debezium-meets-quarkus/

Clement

--
You received this message because you are subscribed to the Google Groups "Eclipse MicroProfile" group.
To unsubscribe from this group and stop receiving emails from it, send an email to microprofile...@googlegroups.com.
To post to this group, send email to microp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/8fd17a1d-d886-481f-9727-9695e4749d14%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.