MicroProfile CQRS specification?

[Edwin Derks]

Hi all,

Is there any need and contribution for a CQRS and/or event sourcing specification for MicroProfile? We talked about it in the data access topic but I think it got side tracked by talking about CQRS and Axon. Therefore, I'm opening this topic about a potentially new MicroProfile specification for CQRS and event sourcing support in MicroProfile.

In the existing discussion, we talked about possible support from AxonIQ, creating a specification based on the Axon Framework with aims of defining and implementing a Java EE / MicroProfile bootstrap next to the current Spring bootstrap for Axon.

Please let me know how you think about this!

Kind regards,

Edwin

[Edwin Derks]

https://groups.google.com/d/msg/microprofile/1sXozNjXLsM/e_8tQAERAgAJ

[James Roper]

It's definitely worth having a discussion about this in its own right.

In my opinion, the basic building blocks for CQRS are:

1. An event log - ie, an abstraction over a database (or multiple different databases) that allows persisting events to a log (eg, a table).

2. A way to emit events on the write side. Event sourcing is one approach, another approach is manually persisting events as part of the same transaction as a CRUD style write side update, and finally another approach is automatic generation of events, either by a framework that understands the updates you're doing, or by database triggers. Not all of these need to be supported, and they could be completely separate, but at least one is needed to do some form of CQRS.

3. A way to consume events on the read side. I put this into two categories:

  a. Streaming events to other services via a message broker - pubsub

  b. Local handlers of events for maintaining local read side projections

I haven't used Axon, but does that align with Axon's approach? It definitely aligns with Akka Persistence's/Lagom's approach.

3a is possible to implement using the MicroProfile messaging spec - the event log just becomes a source of incoming messages that gets processed and passed to an outgoing message broker - this is a use case that we have in mind while creating this spec.

--
You received this message because you are subscribed to the Google Groups "Eclipse MicroProfile" group.
To unsubscribe from this group and stop receiving emails from it, send an email to microprofile...@googlegroups.com.
To post to this group, send email to microp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/302d6f28-83b6-4ae9-9307-fb5df53c5c25%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

James Roper
Senior Developer, Office of the CTO

Lightbend – Build reactive apps!
Twitter: @jroper

[Reza Rahman]

Edwin/James/all,

Apologies for the delay in responding. Rest assured, I was definitely not ignoring your response. I needed to think a little more about this and ask internally on the best way to proceed.

The most useful and productive thing to do is loop in our CTO, Allard Buijze, into this conversation. Allard will properly carry the conversation forward on behalf of AxonIQ. Of course I'll remain engaged as well so please don't hesitate to reach out to me if needed.

Thanks,
Reza

--
You received this message because you are subscribed to the Google Groups "Eclipse MicroProfile" group.
To unsubscribe from this group and stop receiving emails from it, send an email to microprofile...@googlegroups.com.
To post to this group, send email to microp...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/1668c01d-825b-4be7-b033-918d3774c0fd%40googlegroups.com.

[Edwin Derks]

Hi Reza,

That's fine, there was no hurry in getting your response. I just thought it would be best to get the conversation going in its own topic.

I agree that getting Allard involved in our efforts would be great, especially if we are going to proceed with a specification based on Axon's way of CQRS and not for example Akka. The important difference with Akka is of course that Akka uses the actor model, which results in a different way of handling commands and events.

I suggest that we wait for Allard's response in whether or not to join this project, and proceed from there? Honestly, I would love to get this project going but I have not yet figured out how to get it off the ground technically. I hope you guys can assist. :)

Kind regards,

Edwin

[Reza Rahman]

Looping in Allard. I am not quite sure he has had time to subscribe to the alias just yet.

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/cded37f7-3829-4780-bedc-dd25a00d595f%40googlegroups.com.

[Edwin Derks]

Hi James,

Thanks for getting involved as well. :)

I don't know Axon's latest version inside and out but in the version my team is implementing now, Axon handles Commands and persists these in a RDS table (the event store). The EventHandlers are polling for new events in the event store and if they find any, they will read the event data and process it. They keep track of what is the latest event with some form of ID. I will have to check how it works again since I'm not constantly involved in the implementation ;). It sure does not use some kind of message broker there, but of course that doesn't mean our implementations shouldn't do that.

So I think we should see what Allard suggests, and then use best practices from Axon, Akka and Lagom (or similar frameworks), and some of our own insights to proceed with defining a specification?

Kind regards,

Edwin

[Patrik Duditš]

Hi all,

this discussion is interesting to me as I've been looking into event-sourced actors for our Full profile application for past 9 months.

I've been looking for something that integrates easily with e. g. Concurrency utils (which disqualified Akka), and offers concept of Actor-like entity - something that is identifiable, tracks its state changes by means of events but also performs business actions itself. In very rough analogy, something like Event-sourced Entity EJBs :) I didn't find such concept in Axon.

That lead me to creating Event Sourced Entities - https://github.com/goodees/goodees/tree/master/ese/ese-core

At this time is is quite limited, but it was enough to build moderately complex application, but the APIs are flexible enough to support different event stores and execution models as well as e. g. better integration with CDI.

I'm not saying that this is the best model, but I think that Jakarta EE and Microprofile currently lack programming concept that allows to express state and actions "in first person", in single class, and combined with event sourcing this may be quite powerful and yet simple tool.

If you'd like to know more these yet are slides I did for introducing ESE: https://speakerdeck.com/pdudits/event-sourcing-with-ese

Regards,

Patrik Dudits

[allard...@axoniq.io]

Hi all,

it took me a bit of time to find time to respond in between travels and release deadlines...

James' message (https://groups.google.com/d/msg/microprofile/pbYf_7sVKKA/pi1S49g7BQAJ) is quite accurate, although in our opinion, CQRS is as much about Commands and Queries, as Events. But having some good API's in place to publish, store and consume Events would be a decent start.

We would be interested in exploring possible ways for us to contribute in these discussions. Axon Framework has been around for about 9 years now, and there are many lessons that we have already learnt. We're always open to sharing our experiences. 

Let me know how we can be of any assistance.

Kind regards,

Allard Buijze

[Ola Petersson]

I think that this is well worth exploring! In my opinion I think the focus should be on the event-sourcing part and the events (e.g. the append only "log"/db-abstraction, consuming these events etc). CQRS in itself is to me a pattern that you would put on top of this

On Wednesday, June 13, 2018 at 8:16:51 AM UTC+2, Edwin Derks wrote:

[Abhideep Chakravarty]

Totally agree.

[Edwin Derks]

I agree as well

On Wednesday, 13 June 2018 08:16:51 UTC+2, Edwin Derks wrote:

[Emily Jiang]

In my opinion, CQRS and event sourcing were primarily for microservcies data management. It is perfect fit for MicroProfile. It is great that AxonIQ is on board. Why don't one of you draft up the proposal and add to https://github.com/eclipse/microprofile-sandbox?

We can then discuss the proposal in a more constructive way.

Thanks
Emily

[Emily Jiang]

Just wondering who is working on the proposal at the moment. I think a few of you showed some interests on this. I think it is good to do PR on sandbox so that multiple people can contribute and exchange ideas.

Emily

[Allard Buijze]

I have had a brief conversation with James Roper about this. We (i.e. AxonIQ) will happily support any proposal, but at the moment we are unable to take the lead. If I recall correctly, James had a similar situation.

On our side, we are willing and able to contribute by doing reviews, participating in discussions and sharing some designs/api's that we have in Axon at the moment.

Kind regards,

Allard Buijze

--
You received this message because you are subscribed to a topic in the Google Groups "Eclipse MicroProfile" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/microprofile/pbYf_7sVKKA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to microprofile...@googlegroups.com.

To post to this group, send email to microp...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/476b3905-88f0-45a4-9f19-dd0f503051cf%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Allard Buijze
CTO

E: allard...@axoniq.io
T: +31 6 34 73 99 89

[Edwin Derks]

Hi all,

Thanks Emily and Allard for responding. I'm still willing to participate in this project, and if writing a proposal like Emily suggested is a start to get this project going, I might be able to do this. I don't have that much time to participate but if I can write a document that at least contains the big idea of our project, we might have lift off. :)

I will try to deliver a first version within a week and keep you guys updated.

Kind regards,

Edwin

[Emily Jiang]

Thank you Allard! I understand. It is great to have you involved in the proposals and APIs.

Great, thanks Edwin for starting this off! Just do a PR in the sandbox https://github.com/eclipse/microprofile-sandbox?

Thanks
Emily

[Edwin Derks]

Hi all,

Sorry but starting this up is going to take me more time than I initially thought. I considered joining John Clingan's Live Hangout and present the concept of the CQRS specification, but could find the time to properly prepare it on short notice. I will keep it on my radar and hope to deliver something soon, or provide support.

Gr,

Edwin

[Emily Jiang]

Hi Allard/Reza,

In MicroProfile LRA weekly call, we are exploring whether it is possible for Axon to help with LRA. Will either/both of you join the call next week to discuss in more details? The call is on Monday 11:00-12:00 UK time. If the time is not good, we can try to find a better slot.

Thanks

Emily

[m.reza.rahman]

I am afraid I am a bit confused. Axon does not implement LRA (if LRA refers to a way of avoiding full-scale two-phase commit in generic transaction managers involving two or more resources). In fact CQRS and Event Sourcing relies on local transactions and eventual consistency - in a way that is pretty tied to a specific event store (the event store most likely being a single resource). One can say that's the exact opposite of what LRA is trying to accomplish.

How do you see us helping with LRA? What am I missing? Isn't the right people to help application server vendors that already implemented LRA in their generic transaction managers (so folks like Mark Little?).

Sent via the Samsung Galaxy S7, an AT&T 4G LTE smartphone

You received this message because you are subscribed to the Google Groups "Eclipse MicroProfile" group.

To unsubscribe from this group and stop receiving emails from it, send an email to microprofile...@googlegroups.com.

To post to this group, send email to microp...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/4c0d7fbc-153a-45fc-b868-c1429f61e1e8%40googlegroups.com.

[m.reza.rahman]

OK, I am a bit less confused. What I thought you were referring to is LLR. So what is LRA supposed to be? Does not appear to be an establishment term as far as I can see (of course LLR is pretty obscure too so maybe I am just as confused as before)?

[James Roper]

Hi Reza,

I think your understanding of LRA is at least somewhat correct.

The overlap between Axon and LRA is in sagas. They both achieve the same thing, however, LRA assumes a centralised transaction manager that ensures progression of a business operation across multiple services/entities, whereas CQRS sagas use CQRS to progress business transactions. I'm not sure how Axon goes about handling sagas that span multiple microservices, but microprofile-lra does fundamentally handle that.

James

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/5ba1c4ed.1c69fb81.6b4b6.5e41%40mx.google.com.

For more options, visit https://groups.google.com/d/optout.

--

[Tom Jenkinson]

Just a small note that the LRA API is not intended to require a classic centralised transaction manager - indeed one of the goals of the initiative is to develop the API in such a way as to support transports such as CQRS. The LRA specification is really about defining the behaviours that an application programmer can rely upon from the framework (whatever that might be) when coding up applications that require the behaviours of long running actions.

It would be super useful for others who are more familiar with CQRS to help us to meet that goal. One approach would be to compare the annotations present in LRA [1] with the ones that Axon provide [2] but it would be great to get someone who is more familiar with the design of that API to provide some input if some the MicroProfile LRA behaviours would be unworkable in CQRS.

There is then the discussion of how different implementations of LRA can interoperate. There is a discussion on that over here [3] [4] which would be great to get input on.

[1] https://github.com/eclipse/microprofile-lra/tree/master/api/src/main/java/org/eclipse/microprofile/lra/annotation

[2] http://www.axonframework.org/apidocs/2.1/org/axonframework/saga/package-summary.html

[3] https://github.com/eclipse/microprofile-lra/issues/17#issuecomment-421284637

[4] https://docs.google.com/document/d/1HBPiXpsXcQp9T8JeoT-IDs-EblZX1SxuTV5qjUPzwrM/edit#bookmark=id.nqvqtzsk9fjk

To unsubscribe from this group and stop receiving emails from it, send an email to microprofile+unsubscribe@googlegroups.com.

To post to this group, send email to microp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/4c0d7fbc-153a-45fc-b868-c1429f61e1e8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Eclipse MicroProfile" group.

To unsubscribe from this group and stop receiving emails from it, send an email to microprofile+unsubscribe@googlegroups.com.

To post to this group, send email to microp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/5ba1c4ed.1c69fb81.6b4b6.5e41%40mx.google.com.
For more options, visit https://groups.google.com/d/optout.

--

James Roper
Senior Developer, Office of the CTO

Lightbend – Build reactive apps!
Twitter: @jroper

--

You received this message because you are subscribed to the Google Groups "Eclipse MicroProfile" group.

To unsubscribe from this group and stop receiving emails from it, send an email to microprofile+unsubscribe@googlegroups.com.

To post to this group, send email to microp...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/CABY0rKOBHKjuXO-NOnR9Vyj%3DjK7sTEd3gMwfY3aJXasxw5hM_Q%40mail.gmail.com.

[Emily Jiang]

Thank you James and Tom to add more context on this!

Reza, do you have any other questions? Do you think either you or someone you recommend join LRA call next week or weekafter to discuss this subject further?

Thanks

Emily

To unsubscribe from this group and stop receiving emails from it, send an email to microprofile...@googlegroups.com.

To post to this group, send email to microp...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/4c0d7fbc-153a-45fc-b868-c1429f61e1e8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Eclipse MicroProfile" group.

To unsubscribe from this group and stop receiving emails from it, send an email to microprofile...@googlegroups.com.

To post to this group, send email to microp...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/5ba1c4ed.1c69fb81.6b4b6.5e41%40mx.google.com.
For more options, visit https://groups.google.com/d/optout.

--

James Roper
Senior Developer, Office of the CTO

Lightbend – Build reactive apps!
Twitter: @jroper

--
You received this message because you are subscribed to the Google Groups "Eclipse MicroProfile" group.

To unsubscribe from this group and stop receiving emails from it, send an email to microprofile...@googlegroups.com.

To post to this group, send email to microp...@googlegroups.com.

[Reza Rahman]

To be honest, the truth is that neither I nor Allard really have time to attend meetings with too general of a purpose. It's not that we do not mean MicroProfile well (I hope it is abundantly clear that is not the case certainly for me). It is just that devoting time to this means distracting from what we need to do as a small company to keep Axon successful. For example, it means not working on the long pending official CDI support for Axon that I think it a pretty important contribution to the Java EE ecosystem: https://github.com/AxonFramework/cdi.

In addition, honestly I am still struggling to understand the purpose/value of an API so close to CQRS and the saga pattern and is not centered on CQRS and Event Sourcing. With all due respect, it seems a tad dodgy to me at the moment but of course I could be wrong. I do understand the purpose of the project is not standardizing the proven but experimenting with the possibilities.

Here is what I suggest - it sounds like anyone leading the LRA effort should really understand what sagas are in CQRS, ES and Axon. Our documentation on this stuff is pretty thorough and honestly our API on this is pretty dead simple. I would just briefly review that. If there are still questions, please post them here and I will answer them or ask Allard if really needed. Lastly, wasn't there someone else already familiar with Axon on this alias? Couldn't they help with the very basic stuff?

Hope this helps. Apologies for the need to prioritize. Time really is at a premium these days unfortunately. That might change in November or so.

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/aa093526-9a04-41c4-831e-92db264ab1be%40googlegroups.com.

[Martin Stefanko]

Hi all,

I am one of the active members in the MP LRA community and I also have some user experience with Axon and sagas. I'm not familiar with Axon implementation in detail so Reza/Allard can review this afterwards.

As Reza already mentioned above, the Axon's API for sagas is really straightforward [1]. The main idea is that saga is a special kind of event listener which allows the framework to create several instances of the saga class, one for each started saga transaction. The framework then routes events depending on the event type and the property/field value present in the received event which user specified as an association value. The event is than delivered only to the saga instance that contains the event handler for the received event type and is associated with the particular value of the field included in the event. 

I've created a simple example some time ago [2], however, I don't think there has been some major changes in the sagas API since so it should be still sufficient. The saga example is at [3].

Here are some responses to the above question where I may be wrong but I wasn't able to find better solutions by the time I was implementing the above example.

Axon provides ability to handle start and end of the saga on event reception but the actual tracking of the saga execution (dispatched commands) and possible compensation of the saga is up to the developer implementation (user service). This is something LRA handles for the user service transparently.

The communication between microservices can be implemented as the distributed command bus [4]. The responses need to be delivered back to the Axon instance that started the saga as it tracks the saga execution. 

I hope this helps to get the idea.

Also a small note that this thread diverged from the original topic of possible CQRS specification to the sagas / transactions discussion, just a reminder so we don't lose the track of the original subject.

Martin

[1] https://docs.axonframework.org/part-ii-domain-logic/sagas 

[2] https://github.com/xstefank/axon-service

[3] https://github.com/xstefank/axon-service/blob/master/order-service/src/main/java/org/learn/axonframework/orderservice/saga/OrderManagementSaga.java

[4] https://docs.axonframework.org/part-iii-infrastructure-components/command-dispatching#distributing-the-command-bus

[m.reza.rahman]

I'll review the code examples more closely as time permits, but Martin is exactly correct so far. Sagas in Axon really are quite simple by themselves once you understand the basics of CQRS, ES and Axon. There are nuances of how to tweak various lower level things like thread handling but honestly those are probably not that pertinent to an API discussion.

Hope this helps for now.

P.S.: Martin, I would love to get folks like yourself engaged with Axon CDI. Short of outright standardization, I believe it is a step in the right direction for both AxonIQ and the Java EE community. That said, we are still at a very early alpha phase. I will make more of an effort to engage folks myself when I feel the module is ready beyond committed early adopters.

Sent via the Samsung Galaxy S7, an AT&T 4G LTE smartphone

-------- Original message --------

From: Martin Stefanko <xstef...@gmail.com>

Date: 9/26/18 4:52 AM (GMT-05:00)

To: Eclipse MicroProfile <microp...@googlegroups.com>

Subject: Re: [microprofile] Re: MicroProfile CQRS specification?

--

You received this message because you are subscribed to the Google Groups "Eclipse MicroProfile" group.
To unsubscribe from this group and stop receiving emails from it, send an email to microprofile...@googlegroups.com.
To post to this group, send email to microp...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/605ca8be-2aeb-452a-83b4-bedca9544703%40googlegroups.com.

[Michael Musgrove]

The LRA (Long Running Action) specification is a set of CDI annotations for the reliable execution of long running distributed business flows and it is a concrete example of the distributed Saga concept. The reliability aspect comes from the guarantee that compensatory actions will be called if the LRA is cancelled.

Tom Jenkinson gave a sanppy summary of how LRA, CQRS and Event Sourcing are related to each other:

"Implementations of MicroProfile LRA can conceivably be implemented within a CQRS framework. The sagas themselves do not specifically require CQRS/event-sourcing as per https://www.cs.cornell.edu/andru/cs711/2002fa/reading/sagas.pdf."

Although there is a Java client for controlling LRAs the specification is meant to be a CDI first approach and it defines annotations that controlling the lifecycle of an LRA and for controlling participation in the LRA. Each spec implementer is free to choose whatever engineering solution they deem most natural, so, for example, I would expect Axon would map the spec onto their own system by generating whatever events are appropriate.

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/36622b5e-985a-63d0-57ea-2f25bf0194e2%40gmail.com.

[Mark Little]

+1 definitely the right approach.

> To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/CAKJ8-j5BM7_kjRDd9fCKQTbai77-xsN9Qwj6vC5jTPm6GQq67Q%40mail.gmail.com.

[Emily Jiang]

In last week's MicroProfile BOF (Wednesday), we discussed MicroProfile LRA and its issues. Fortunately, Reza was there and happy to help out. Reza planed to do a detailed discussion on Axon usage etc on the following day (last week). Since there were not enough people showing up, Reza kindly agreed to do a hangout and open to the MicroProfile community. Reza, would you like to start a doodle pool to get a convenient time slot?

Thanks

Emily

[Mark Little]

Good but can someone summarise afterwards so those who cannot participant can chime in here? Also needed for future references to this archive.

Mark.

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/c389e464-426c-40c5-8284-06909baa5892%40googlegroups.com.

[m.reza.rahman]

I think it is best someone more intimately familiar with the workings of the group set up the meeting. Otherwise, I will be happy to. Besides I honestly think someone else other than me needs to moderate anyway.

I am generally available 9 AM - 9 PM EST including on the weekend. My schedule should be adaptable to what has worked best for the group. I would advise one and a half or two hours for this meeting. The first hour for an introductory presentation and half hour or an hour for subsequent discussion. An alternative is just the introductory talk first and then having a discussion on a subsequent meeting.

Once the meeting is set up, I can help spread word. It appears to be a popular topic so it may be well attended (and could be broadcast/recorded via Hangouts on Air).

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/c389e464-426c-40c5-8284-06909baa5892%40googlegroups.com.

[Emily Jiang]

thanks, Raza, I will start a doodle pool to find a best slot for the interested party especially LRA workgroup to attend.

Here is the doodle pool https://doodle.com/poll/yua8hzrdw4rcrc6r. Please put your availability there. We can start with one slot and then agree on the second slot afterwards.

I'm hoping to have this call next week. The pool is open till Friday and I will announce the slot over the weekend.

Thanks

Emily

[Marcia Peterson]

Emily,

1.  I really like the ideal of using the doodle pool to raise awareness for the projects. Also, you might want to make the Project Name in Bold and Underlined when sending out request to avoid confusion.  We (Debbie Hoffman, Edwin Derks, and Marcia Peterson) need a doodle pool to raise awareness to the Wikki Project. Could you please, start one. I don't want to have any confusion with the different projects going on so, would it be ok, for you to be the doodle ette. - NO PUN INTENDED! Or, maybe you don't mind someone having access as well, please let me know.
   

Thank you,

Marcia Peterson

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/9c348227-dadf-495f-9bae-ff6d6510b475%40googlegroups.com.

[Emily Jiang]

Thank you Marcia! I have responded to the corresponding thread and provided the instructions on creating doodle pool. Hope that helps!

Thanks

Emily

[Marcia Peterson]

Thanks, for your help your a gem.

Marcia Peterson

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/a51ee710-1c79-40c0-ad92-187177c3503a%40googlegroups.com.