MicroProfile Event Sourcing Proposal

144 views
Skip to first unread message

Gordon Hutchison

unread,
Jun 9, 2019, 6:03:11 PM6/9/19
to Eclipse MicroProfile

Hi MicroProfilers,

I would now be possible to build on top of the coming MicroProfile Reactive Messaging specification
and splice in change data capture from something like Debezium together with application level support for change event
production, outboxing, distribution and handling from lumps that mostly exist already.

Distributed microservice data consistency is a big customer concern and event driven eventual consistency is
a very common pattern where we can add value.

There are a number of user and vendor advantages to doing this under a MicroProfile standardisation but
not as a straight continuation of the either MicroProfile Reactive Messaging, Apache Kafka or the Debezium projects.
See the alpha proposal draft linked below for more details.

I have had some preliminary discussions with Gunnar Morling and Clement Esscofier.
It seems to make sense to us to do the groundwork to prepare a proof of concept and perhaps a poposal for a
MicroProfile Event Sourcing Specification.

There are the beginings of a sketch of this here:
created under the current MicroProfile process here:

You will see in this PR:
the very very start of a PoC working party to build bits of prototype at

If you are interested in this area and feel you have something to contribute at this
boot-up stage (or at any stage) then please feel free to chip in or get involved.

3 days in, we have an open door and have not yet self-organised roles or process and it
it will be some time before we submit the proposal for consideration, but
you know what they say - feedback early, feedback often
so please share any of your thoughts on the topic.

Gordon.

Ken Finnigan

unread,
Jun 10, 2019, 9:12:58 AM6/10/19
to MicroProfile
Gordon,

I think for now it's best to focus the work in https://github.com/smallrye/smallrye-event-sourcing to develop the API and implementation, then once all parties are happy with how the API is defined we can put forward a complete MP spec proposal in the sandbox.

Please submit a PR with the current doc in sandbox to the above repo and we can start working on it there.

Ken

--
You received this message because you are subscribed to the Google Groups "Eclipse MicroProfile" group.
To unsubscribe from this group and stop receiving emails from it, send an email to microprofile...@googlegroups.com.
To post to this group, send email to microp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/a8b24578-646a-4cf8-83f2-3a7d0292c300%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Gordon Hutchison

unread,
Jun 10, 2019, 9:37:31 AM6/10/19
to Eclipse MicroProfile

Hi Ken,
OK I will do the smallrye PR to transfer over the files as they are
and stop making any changes in the MP sandbox for now.

Gordon.

Ken Finnigan

unread,
Jun 10, 2019, 9:41:55 AM6/10/19
to MicroProfile
Thanks!

--
You received this message because you are subscribed to the Google Groups "Eclipse MicroProfile" group.
To unsubscribe from this group and stop receiving emails from it, send an email to microprofile...@googlegroups.com.
To post to this group, send email to microp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/88348da3-96f2-4a77-8b9f-5c618f8ba197%40googlegroups.com.

Gordon Hutchison

unread,
Jun 11, 2019, 5:47:52 AM6/11/19
to Eclipse MicroProfile
Actually as I had already put the link to the sandbox doc in
the MicroProfile hangout agenda and forgot a paragraph on
change event inversion to do undo/compensation I added that in both places.

Gordon Hutchison

unread,
Jun 12, 2019, 4:14:06 AM6/12/19
to Eclipse MicroProfile
Added a paragraph on the possibility of integrating with Cloud Events via the MicroProfile Reactive Messaging layer
rather than as expressed already (and where I got the idea from (Yuchao Wang?) ) in Debezium: https://issues.jboss.org/browse/DBZ-1292

Gordon Hutchison

unread,
Jun 25, 2019, 3:11:54 PM6/25/19
to Eclipse MicroProfile



Gordon.

Gosling Von

unread,
Jul 12, 2019, 4:08:00 PM7/12/19
to microp...@googlegroups.com
Hi guys,

I've been following the direction of the event & messaging & streaming in EE. I have skimmed the proposal in Github[1], It seems that Event Sourcing original intention is similar to my founded another project OpenMessaging Connector[2]. 

So, I take the liberty of asking, where shall we discuss the Event Sourcing, I would like to get involved in and figure out could we work together in this direction  :-)





Best Regards,
Von Gosling

On Jun 26, 2019, at 3:11 AM, Gordon Hutchison <gordon.h...@gmail.com> wrote:




Gordon.

--
You received this message because you are subscribed to the Google Groups "Eclipse MicroProfile" group.
To unsubscribe from this group and stop receiving emails from it, send an email to microprofile...@googlegroups.com.
To post to this group, send email to microp...@googlegroups.com.

Gordon Hutchison

unread,
Jul 16, 2019, 9:13:13 AM7/16/19
to Eclipse MicroProfile


Hi Von Gosling,

Great to see your interest in the Event Sourcing project
and apologies for not replying sooner - I am on jury duty at
the moment so internet access is a bit lumpy.

I will study your Open Messaging project :-)

We don't currently have a verbal discussion on the event sourcing project.
Communication is currently via issues (feel free to create or comment!) or on https://gitter.im/smallrye-io/event-sourcing.

With reference to microprofile reactive messaging, I can't better the answer from John Clingan here:

Another possibility is putting something on the MicroProfile community hangout adgenda -
if you want to advocate verbally for open messaging slotting into MicroProfile.

for the link. I am sure you would be very welcome.

Gordon



On Friday, 12 July 2019 21:08:00 UTC+1, Gosling Von wrote:
Hi guys,

I've been following the direction of the event & messaging & streaming in EE. I have skimmed the proposal in Github[1], It seems that Event Sourcing original intention is similar to my founded another project OpenMessaging Connector[2]. 

So, I take the liberty of asking, where shall we discuss the Event Sourcing, I would like to get involved in and figure out could we work together in this direction  :-)





Best Regards,
Von Gosling
On Jun 26, 2019, at 3:11 AM, Gordon Hutchison <gordon....@gmail.com> wrote:




Gordon.

--
You received this message because you are subscribed to the Google Groups "Eclipse MicroProfile" group.
To unsubscribe from this group and stop receiving emails from it, send an email to microp...@googlegroups.com.

Allard Buijze

unread,
Jul 17, 2019, 3:10:31 AM7/17/19
to Eclipse MicroProfile
Hi all,

I've been watching this mailing list for a while now, and have also been involved in some discussions/calls in the past to explore CQRS / ES related topics. I have a few doubts/concerns that I'd like to share regarding the Event Sourcing proposal.

First, a bit of background: I've been actively using CQRS and Event Sourcing for 10 years now. In that time, I've developed Axon Framework [1], which provides APIs and implementations of building blocks required to build applications based on the concepts of DDD, CQRS and Event Sourcing. I'm also an active member of the DDD community.

The main concern I see is the use of the term "Event Sourcing". In most places, the term "Event Sourcing" is used, while the actual approach described is Event-Driven or Event Processing. The main difference is in how the publisher of events uses those events for its own state. If a component generates events, and another component uses those to build up some state, it's Event Processing. It becomes Event Sourcing, when the entire system, including the publishing component, uses these Events to build up its state. If you can delete all data, except the events, and the system will automatically (possibly after a long initialization) continue processing normally, then it's Event Sourcing.

Some other concerns I see, but those are somewhat more subjective, is the suggestion to "automatically" generate events off of state differences (while still using the stored state for making new decisions). Martin Fowler gave an interesting presentation at Goto Chicago in 2017 [2], where he described that Events are a "First class thing in the gig". Practices such as Event Storming and Event Modelling are gaining momentum very quickly because they prove to be very effective in describing and designing complex processes, using the Event as a core design element. I have seen many systems being built (and have also been involved in building some of them, unfortunately) where events were generated as side-effects. In reality, the resulting events are hardly every a truthful, complete, representation of the history of an application. Most importantly, the "intent" is lost, as events only describe data deltas. In "Jury duty" terms: if the events don't give you the truth, whole truth and nothing but the truth, their use is severely limited.

My callout to the Microprofile community would be to not start from scratch. There is a lot of experience with CQRS and Event Sourcing out there. Most mistakes have already been made, and the good parts have already made their way to frameworks and libraries, such as Axon and Lagom.

Just my 2cts.....
Cheers,



On Tuesday, 16 July 2019 15:13:13 UTC+2, Gordon Hutchison wrote:


Hi Von Gosling,

Great to see your interest in the Event Sourcing project
and apologies for not replying sooner - I am on jury duty at
the moment so internet access is a bit lumpy.

I will study your Open Messaging project :-)

We don't currently have a verbal discussion on the event sourcing project.
Communication is currently via issues (feel free to create or comment!) or on https://gitter.im/smallrye-io/event-sourcing.

With reference to microprofile reactive messaging, I can't better the answer from John Clingan here:

Another possibility is putting something on the MicroProfile community hangout adgenda -
if you want to advocate verbally for open messaging slotting into MicroProfile.

Reza Rahman

unread,
Jul 17, 2019, 12:59:39 PM7/17/19
to microp...@googlegroups.com, Allard Buijze
To be honest, I agree with Allard. I currently just don't have the bandwidth to get too involved in this, especially since I prefer to be involved in Jakarta EE instead. Surely there is someone else aside from me and Allard with a bit more experience with Axon that can help out here?

Reza Rahman
Principal Program Manager
Java on Azure

Please note that views here are my own as an individual community member and do not represent the views of my employer.

P.S.: Do we really need to get into this area in MicroProfile at all? I had started work on a CDI extension for Axon. Maybe the effort expended here could simply be spent maturing the CDI extension for Axon?
To unsubscribe from this group and stop receiving emails from it, send an email to microprofile...@googlegroups.com.

To post to this group, send email to microp...@googlegroups.com.

Gordon Hutchison

unread,
Jul 18, 2019, 8:36:55 AM7/18/19
to Eclipse MicroProfile


Hi Reza and Allard,


Interesting to read your responses


Allard, if your main concern is the use of the term 'Event Sourcing' 

– then I actually read that as a very positive sign. :-) 

Naming is hard and the area of event/message approaches 

is very cluttered with existing terminology.


'Event sourcing', though not ideal as a name, seems to work well as a ‘flag’ at the moment to anchor the area of interest – 

even if domain experts can correctly point out the rattle in the fit with previous uses of the term.

 

We don’t currently know exactly where the smallrye os project will go and if it turns out that there is

a better name for what it ends up being then there is a good checkpoint to a the better name when/if 

we actually write a proposal for MicroProfile (if we ever do).


The categorisation of approaches that you refer to from Martin Fowler is all about 

styles of application design. We are not really in the business of application design 

or even frameworks that constrain or guide users’ application design approach. 


As I see it, we are in plumbing supply business.


If, as you say, one is to define Event Sourcing as having even the event creators relying on events 

to drive 100% of their state changes – as far as I know, none of the intent of anything in the SmallRye event soucing 

project so far speaks against that.  User’s may be aware of Greg Young’s advice to ‘don’t put business 

logic between an event and the storing of that event’. That is application design advice and if in following it, 

“the publisher of events uses those events for its own state“, one might arrive at an event sourcing style by your definition.


So does this design approach really lead us ‘away’ from using reactive streams or messages/events for the underlying plumbing

as we propose?


I would argue no. If we wanted to split off bit of application function, that was event driven, 

to store the application state (change) - reactive streams provides an excellent abstraction for doing this 

(argument here: https://groups.google.com/d/msg/microprofile/18SNnnWe4pQ/6KeoL7nIGgAJ )


Microprofile reactive messaging ‘channels’ also benefit from local/remote transparency – 

so splitting  a service so that all ‘state’ is handled downstream of an event 

(the event could be persisted by a processor/operator very cleanly) can be done either all inside one service/JVM process or 

switched to be across a Kafka queue dependant on channel config. 

I would be very interested if there was an even better set of ‘plumbing’ suitable for 

supporting an Event Sourcing (as traditionally defined) style?


In terms of ‘not starting from scratch’ – that is certainly good advice no one would argue with. 

We get a huge amount of benefit starting from where we are – MicroProfile Reactive Messaging 

which in turn exploits Reactive Streams Operators and its reactive programming stack. 


As well as a lot of the wisdom and material from yourselves, Greg Young, Chris Richardson, 

Marin Fowler etc. We were very lucky to benefit from this thread last year:

https://groups.google.com/d/msg/microprofile/FdF-W8zEW34/VZtOGHfHAgAJ

and these talks on Axon from Reza https://www.youtube.com/channel/UCf-SCs0ylgqVIWfNElPEZhw . 

We also discussed if MicroProfile could support a

CQRS framework in the thread around https://groups.google.com/d/msg/microprofile/pbYf_7sVKKA/EkAPU7RdCgAJ

(interesting to review James’ ‘basic building blocks’ there).


Forgive my blunt and grossly inaccurate language in this:….have we already been around 

the loop of something like Axon becoming a template for a MicroProfile CQRS/ES/Saga approach? 

I can remember spending quite a number of hours on calls discussing this (links above).  

What has or would change to make it likely that we would have a different result if we ran that again?


Also – what would the value add to Axon be in the MicroProfile standardisation of bits of it – 

it is not like we would have multiple implementations from different vendors? 

Do you really want an IBM or Redhat Axon Framework?


Personally, I am not yet convinced that the MicroProfile community is ready 

to support standards work for something that high up the stack – a

n app framework narrows the applicability to microservices that ‘fit’ the framework’s prescription. 

Others may have a different take of course.


I would seek to find the right balance between


  1 – continuing to make the MicroProfile reactive assets/vision more valuable

  2 – benefitting from ‘other’ technology/wisdom (Axon, Eventuate, Debezium etc.)

  3 – not becoming over opinionated about application design


If you have a positive, literal, suggestion for a better name for the https://github.com/smallrye/smallrye-event-sourcing project (or anything else)

please raise an issue at https://github.com/smallrye/smallrye-event-sourcing/issues and we can process it. 

In terms of naming we had thought of ‘Change Events’ based on CDC - but want to the project to really not be all CDC based. 

Also ‘Change Events’ tends to lose some of the intention and context inherent in many microservice events.  

‘Event Driven’ – well we are not app code. Event Driving – well it also is about handling etc etc.

Naming is hard.


It would be great to benefit from your experience in an ongoing way – so please (anyone) what do you think about

the above?


Gordon.  

Reply all
Reply to author
Forward
0 new messages