Library of shared Connectors to plug-into MicroProfile Reactive Messaging as Stream sources/sinks

[Gordon Hutchison]

Hi Folks,

I observe that other platforms often have a library of 'Connectors' to connect them to different sources and sinks of data:

For example:

   https://doc.akka.io/docs/alpakka/0.18/connectors.html

With the MicroProfile Reactive Messaging Connector SPI taking shape here:

https://github.com/eclipse/microprofile-reactive-messaging/tree/master/api/src/main/java/org/eclipse/microprofile/reactive/messaging/spi

and the fact that most MicroProfiler implementors probably want to commect to a

roughly similar set of technologies underneath each of which has a client API

This opens up the possibility of a shared library of MicroProfile Connectors that plug

in as reactive stream sources/syncs to the MicroProfile ecosystem.

Connectors are being specc'ed to be classes discovered using CDI and are bridging from

a known SPI to a (one of a set of) known client libraries. If we can implement them using only

MicroProfile portable code then they could probably be shared across implementations.

Perhaps this could go into the SmallRye project?

A good set of Connectors as reactive stream sources and syncs would help in the adoption

of MicroProfile as a stronger reactive platform when comping to alternatives and help

our customers connect to their existing applications.

Any thoughts?

Gordon Hutchison

[Gordon Hutchison]

I thought I would have got a response along the lines of

"this already exists" by now :-)  at the subdirs at:

https://github.com/smallrye/smallrye-reactive-messaging/smallrye-reactive-messaging-*

But I am wondering about the concept of factoring out from those

the core specific ascpects.

So for example,

https://github.com/smallrye/smallrye-reactive-messaging/blob/master/smallrye-reactive-messaging-cloud-events/pom.xml

and associated code could be forked and factored out to contain dependancies that would be resolvable from any MicroProfile

implementation so that the connectors could be more portable cross-platform.

Gordon.

[Gunnar Morling]

Hi Gordon,

That's an interesting idea. One thing we could explore is integration of CDC connectors like Debezium as sources into Reactive Messaging.

We had discussed it in the context of RxJava some time ago with Clement; but it didn't get very far back then, mainly due to the fact that CDC connectors are stateful (they must remember their last processed source offset, so they can continue from there after a restart). Surely nothing unsurmountable, but we could need some advice/help from folks more knowledgeable in the reactive domain.

If you're interested, it's something we could PoC together (Debezium has a mode where it can be used as a library without requiring Kafka or another messaging broker), which would be a good fit for this usage as Reactive Messaging source).

--Gunnar

[Gordon Hutchison]

Hi Gunnar, and others

Very interesting reply Gunnar thank you, I do think there are possibilities here for adding real customer value.

As I presume you know, MicroProfile Reactive Messaging uses the 'shape'

(signature and return type) of the @Incoming and @Outgoing

methods to imply how to wire them up as stream Subscribers/Publishers.

It had occurred to us that there could be a convention based on an

@Incoming method of a particular class type (a domain type or a generic over a domain type)

that would act as a 'catcher' for wiring into the Event Sourcing source topics.

In terms of sending events - one could easily envisage a sender (@Outgoing) that

had the level of consistency that is offered, for example, by Axon (if my understanding

is correct).  But did you have a picture already of how this could be strengthened to

offer the local transaction commit gateway that the outbox pattern and Debezium

provide? (Of course this is at the Java/application method level and not CDC by plugging

into the database.)

I return, in my head, to what it is the customer value of this not just being a delta on

Axon or Debezium?  

It is certainly possible to put a local (or even XA)  'commit' into the wiring between a local @Outgoing

and a remote @Incoming connected across Kafka events.

What comes up is heterogeneous interoperability of Event Sourcing

between microservices built with MicroProfile but on different technologies/vendor platforms for

each microservice.

Was the PoC you were thinking of along these lines or the half that is a 'catcher' of Debezium sourced

CDC kafka events built using MicroProfile Reactive Messaging?

In other words, in 

"Debezium as sources into Reactive Messaging" were those seperated by Kafka i.e.

1) a Debezium Kafka Event understanding remote Reactive Messaging @Incoming

or

2) a generic Reactive Messaging @Outgoing Kafka Publisher fed by a local Debezium system utilising a DB's CDC plugin-mechanism?

In terms of working together on a PoC - that's what MicroProfile seems to be great for.

I can't say I own my own time to spend (IBM does :-) ) but I can see the clear customer value in having the

abiltity to connect different CDC/EventSourcing microservices together

(so I would be hopeful about getting further time to spend on it)

and this is a great fit to MicroProfile Reactive Messaging.

I would be very interested in opinions from people involved in the other MicroProfile implementations

#usualsuspects and from Axon etc.

Once we want to take this off to more detail (not yet probably) we could use the MicroProfile Reactive Messaging

github repo to raise individual discussion issues? Or somewhere else?

Fascinating as always Gunnar.

Gordon.

PS.

Is there a term to 'brand' or refer to  "local-outbox-commit-event-catcher-eventual-consistency" I tend to think

of CDC and EventSourcing and Eventual consistency as too long and CQRS implies something

of the application structure?  I invite anyone to propose a shorter name :-) and we can vote on it.

[Gordon Hutchison]

By the way Gunnar, I did get that you were pointing at:

@Incoming( "mysql-inv-customer" )

@Outgoing( "kafka-event-feed" )

public Message<MySqlServerInventoryCustomers> process(MySqlServerInventoryCustomerCDCEvent event) {

    return Message.of( packageForEventSourcing(event));
}

With a config like:

mp.messaging.incoming.mysql-inv-customer.connector=debezium.cdc.mysql
mp.messaging.connector.debezium.cdc.mysql.connector.class=io.debezium.connector.mysql.MySqlConnector
mp.messaging.connector.debezium.cdc.mysql.database.hostname=192.168.99.100
mp.messaging.connector.debezium.cdc.mysql.database.port=3306 
mp.messaging.connector.debezium.cdc.mysql.whitelist=inventory 

...
mp.messaging.outgoing.my-channel.connector=acme.kafka
mp.messaging.outgoing.my-channel.bootstrap.servers=localhost:9096
mp.messaging.outgoing.my-channel.topic=my-topic

(copy/pasting (badly) from https://debezium.io/docs/connectors/mysql/#events

:-)

I would like to add on top of this a convention for the topic naming and event catching

using MicroProfile Reactive Messaging @Incoming.

In order to create a MicroProfile Event Sourcing solution.

(As well as the very clever debezium CDC of course it would be 'simple'

to have a plain old @Outgoing at the application level where the Message type was a

EventSourcingMessage.

Gordon

On Wednesday, 29 May 2019 20:43:00 UTC+1, Gunnar Morling wrote:

[Gordon Hutchison]

In the above

mp.messaging.outgoing.my-channel.connector=acme.kafka
mp.messaging.outgoing.my-channel.bootstrap.servers=localhost:9096
mp.messaging.outgoing.my-channel.topic=my-topic

"my-channel" should be "kafka-event-feed"

[Gordon Hutchison]

https://github.com/eclipse/microprofile-sandbox/pull/53

Gunnar ^^^

Clement    ^^^

On Wednesday, 29 May 2019 20:43:00 UTC+1, Gunnar Morling wrote:

[Gunnar Morling]

Hi Gordon,

Sorry for staying silent, still on my todo list to get back to your earlier mail. But yes, what you sketch here is exactly what I have in mind: having ingoing channels for CDC connectors, which would feed data change events right into the application.

--Gunnar

--
You received this message because you are subscribed to a topic in the Google Groups "Eclipse MicroProfile" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/microprofile/F0ehhd1MFMc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to microprofile...@googlegroups.com.
To post to this group, send email to microp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/722bbacf-122e-4088-84c5-8f4b11d4f81f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Gordon Hutchison]

OK great, glad to be on the same page Gunnar. In terms of "If you're interested, it's something we could PoC together."

I'd be well up for that (woven into schedules for getting Reactive Messaging going on OpenLiberty of course :-) ).

Ken Finigan is suggesting we could use a Smallrye repo "to start experimenting with API/Impl that can then feed into a spec".

I know Clement had some thoughts too and is used to using SmallRye.

How does that sound for you?

Gordon

On Thursday, 6 June 2019 16:59:57 UTC+1, Gunnar Morling wrote:

Hi Gordon,

Sorry for staying silent, still on my todo list to get back to your earlier mail. But yes, what you sketch here is exactly what I have in mind: having ingoing channels for CDC connectors, which would feed data change events right into the application.

--Gunnar

To unsubscribe from this group and all its topics, send an email to microp...@googlegroups.com.