Data access spec for MicroProfile

[Nathan Rauh]

Is there any interest/support in the community for defining a data access spec for MicroProfile?

After examining other data access technologies, we could re-use and improve on a number of concepts.

Concepts to reuse:
 - Ability to represent data as POJOs
 - Provide boilerplate template interface that can be extended by the user (for example, similar to Spring Data Repository)
 - Inject boilerplate or user-defined interfaces using CDI
 - Provide a common layer of abstraction above existing technologies

Concepts to improve upon:
 - Define operations in a simple and type-safe way using annotations that are clearly JavaDoc'ed instead of the user needing to name methods a certain way
 - Allow for a simplified query language on method level annotations to express more complex operations than what can be handled by parameter level annotations
 - Use MP config and/or annotated POJO to define configuration

Goals would be:
 - Standardize and greatly simplify basic patterns for data access
 - Define an API that works for both relational and NoSQL database
 - Avoid the need for the user to write boilerplate code, and the need to manually manage life cycles of objects


For example, a predefined template interface could have basic generic methods such as these,

public interface DataAccessTemplate<K, T> {
    public boolean contains(T item);
    public boolean containsKey(K key);
    public T findBy(K key);
    public K insert(T item);
    public boolean insert(K key, T item);
    public long remove(T item);
    public boolean removeBy(K key);
    public long update(T item, T updatedItem);
    public boolean updateBy(K key, T updatedItem);
    ... more should be added
}

The default template could be injected and used directly (in this case, the user does nothing but supply POJOs and use the built-in template),

@Inject @Data
DataAccessTemplate<String, Customer> customerdb;

Customer c = customerdb.findBy(customerId);
c.setShippingAddress(...);
customerdb.update(c);


Optionally, the default template could be extended with spec-defined annotations to create custom queries.

For example, given the following POJO,

public class InventoryItem {
    public String productId;
    public String productName;
    public String description;
    public long quantity;
    public float weight;
    public boolean includesPackaging;
    public float length, width, height;
}

some custom queries might be,

@Data(key = "productId")
public interface InventoryDB extends DataAccessTemplate<String, InventoryItem> {
    boolean decrementQuantity(@Equal String productId, @Decrement("quantity") @GreaterOrEqual("quantity") long amount);

    @Sort("weight")
    @Where("length * width * height > :maxVolume")
    Stream<Inventory> findItemsWithExtraShippingCharge(double maxVolume, @Greater double weight);

    @Update(attribute = "weight", value = "weight + :w * 2 * (length * width + width * height + height * length)")
    @Update(attribute = "includesPackaging", value = "true")
    @Where("includesPackaging = false")
    long addPackagingWeight(float w);
}

Example usage,

@Inject
InventoryDB inventorydb;

if (inventorydb.decrementQuantity(prodId, amountToPurchase)) {  // custom method
    ...
} else {
    InventoryItem item = inventorydb.findBy(prodId);  // built-in method from DataAccessTemplate
    out.println("Only " + item.getQuantity() + " of " + item.getProductName() + " are in stock");
}

Annotations at the parameter level, such as @Equal/@Greater/@GreaterOrEqual in the above example provide a convenient way to indicate conditions on attributes that must be met for a simple query.  Conditions are ANDed together such that all conditions must be satisfied.
Annotations at the parameter level, such as @Decrement in the above example, provide a convenient way to indicate mutation of attributes.
The annotation value optionally identifies the attribute name (useful if different from the parameter name, which it otherwise defaults to) upon which the comparison or operation is performed.  In the example, note the "quantity" attribute name identified on both @Decrement and @GreaterOrEqual which indicates that these apply to the POJO attribute 'quantity'.  The 'quantity' attribute will be decremented by 'amount' if both conditions are met: that the 'quantity' is greater than the 'amount' and the 'weight' attribute is greater than the supplied 'maxWeight' value.

Annotations at method level, such as @Where and @Update allow more complex queries and assignments.
Method parameter names are usable as variables within the expressions.  Attribute names from the POJO can also be used in these expressions.  For this, a basic query language is needed.  An easy option (used for the example) would be to borrow a core subset of JPQL which is already a standard and which is modeled after SQL so has a good chance of being familiar.  The goal would be a basic set of common operations and comparisons that could hopefully be intuitive and clearly defined within a single page or two of JavaDoc.

All of this is kept free of particular syntax or API calls specific to backend databases or technology types, leaving the container/appserver implementation free to be done in a variety of ways.  For example, it might involve implementing on top of JPA for relational or JNoSQL for non-relational or maybe implementing specifically for certain database type(s) such as MongoDB or MySQL, and so forth.

We are interested to know the community's thoughts on a data access spec such as this.  If we can get a general consensus that it would be valuable, we would like to begin pushing some more detailed proposals to the microprofile-sandbox repo.

Thanks!

[Kevin Sutter]

Thanks, Nathan, and welcome to the MicroProfile community!

Can you clarify something for me?  You indicated "All of this is kept free of particular syntax or API calls specific to backend databases or technology types...".  So, you are proposing this Data Access component without introducing new dependencies on JPA or things like JNoSQL?  Limiting the hard dependencies is a goal of both MicroProfile and Jakarta EE.

Also, you referenced Spring Data below...  Is this just migrating Spring Data to MicroProfile?  Can you further expand on the relationship to Spring Data, or contrast with Spring Data capabilities?

I have lots more questions, but let's start with these to get the conversation going...  :-)

Thanks, Kevin

[Andy Guibert]

Hi Kevin,

I've been brainstorming ideas with Nathan before he sent this proposal, so I'll try to chime in and answer your questions.

> So, you are proposing this Data Access component without introducing new dependencies on JPA or things like JNoSQL?  Limiting the hard dependencies is a goal of both MicroProfile and Jakarta EE.

Correct, the basic idea is that we would define a generic data access API that only had a dependency on CDI (and perhaps MP Config).  It would be up to the vendor implementation what technologies they implemented on top of.  Similar to how JPA doesn't mandate that you need to work with Oracle, DB2, MySQL, etc.  So basically implementers of this spec could pick and choose which backend DBs they support.

> Also, you referenced Spring Data below...  Is this just migrating Spring Data to MicroProfile?  Can you further expand on the relationship to Spring Data, or contrast with Spring Data capabilities?

Spring Data has shown that even JPA can be abstracted to a higher level technology.  We're not suggesting that we migrate Spring Data to MicroProfile (as that would drag a lot of other dependencies along with it).  Rather, we are trying to take the parts that work well, and improve on some parts. 

To directly compare and contrast this proposal with Spring Data...

Similarities:

  - allow user to write custom interfaces which define the type of POJO to operate on, and various select/update operations

  - provide a boiler-plate interface that covers the golden path use cases

  - standardize and simplify configuration (i.e. no need for persistence.xml)

Differences:

   - use annotations (with javadoc) instead of the user needing to name methods a certain way, for example:
      // Spring Data

      List<Inventory> findByWeightGreaterThanOrderByWeightAsc(double weight);

      // this proposal

      @Sort(value = "weight", ascending = true)

      Stream<Inventory> findItemsWithExtraShippingCharge(@Greater double weight);

   - there are different branches of Spring Data (Spring Data JPA, Spring Data MongoDB, Spring Data JDBC, etc).  In this proposal we hope to have a single API that feels the same regardless if it's built on top of JPA, JDBC, JNoSQL, etc.

   - the rest of the "Concepts to improve upon" section

Hopefully that answers more questions than it raises =)

Thanks, Andy

[James Roper]

One of the challenges in microservices (which MicroProfile is intended to serve) is that because you have your data spread across multiple services, each using their own database, you lose ACID transactions on your updates. This makes plain CRUD operations problematic, because if you need to do two different CRUD updates in two different services, there's no way to guarantee atomicity, no way to ensure that uncomitted reads don't occur, etc. In monoliths, ACID transactions allowed us to take consistency for granted, to the point where we forget that we even needed consistency, in spite of consistency being critical to the correct functioning of the systems we were writing. This has caused a problem in the move to microservices, because people have forgotten how important consistency is, and they don't realise how much they relied on it, so they plough on ahead using the same approaches to persistence that relied on ACID transactions, except now they don't have them anymore, and so they have problems.

I think a data access API designed in 2018 must address this as a first class issue. I don't think 10+ year old approaches like the one being proposed here is a step forward for MicroProfile, rather, it's something pegging MicroProfile to the past.

There are a number of ways to address this. The two ways that we've looked at in depth at Lightbend are:

1) Event sourcing. Rather than storing the current state, store the events that occur in an event log, and compute the current state from the events. By using event sourcing, you can then publish your events to a message broker without needing transactions, by consuming your own event log in a way similar to consuming a Kafka queue, you track what offset you have published, and if you encounter a failure, you restart publishing from the last offset that you successfully published - thus you have guaranteed delivery to the message broker, which you don't have when you just publish to the message broker at same time as doing an update. This can be used and take advantage of at least once messaging to propagate events to other services that need them, thus ensuring eventual consistency (as opposed to no consistency guarantees when using just vanilla CRUD). Event sourcing is the primary persistence API that we offer for Lagom.

2) Complementary event log. In this, you have a traditional CRUD schema, but with every update you do to your schema, you also persist an event to the event log in the same transaction. This event log can then be published to a message broker without needing transactions, and updates can be propagated with at least once messaging guarantees to other services. This has the downside of having two sources of truth (the event log, and the CRUD schema) and what if they disagree? But it does solve the consistency guarantees issue. An API that supports this might provide an API where committing a transaction is done by emitting one or more events to the event log.

In my opinion, if we are to provide a new API for persistence in MicroProfile, it must support an event log. I think ideally it should support event sourcing, but the complementary event log use case could also be supported.

--
You received this message because you are subscribed to the Google Groups "Eclipse MicroProfile" group.
To unsubscribe from this group and stop receiving emails from it, send an email to microprofile...@googlegroups.com.
To post to this group, send email to microp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/7aa8ced6-d969-4bfb-8853-12db86a08b7b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

James Roper
Senior Developer, Office of the CTO

Lightbend – Build reactive apps!
Twitter: @jroper

[Emmanuel Bernard]

Hey James,

While I am sympathetic to the idea of event sourcing, my experience has been that it is a relatively high bar for teams and I don't think it should be the only model. Event Sourcing is not a bunch of inserts and updates put in a log as you know and of a much higher level business meaning than what Java people call entities.

Also, I've seen and pushed a model where indeed data changes and provided as input to a microservice. But that does not preclude the said microservice to have an actual datastore underneath and thus wanting some more traditional data access technologies.

Anyways, that's a brief version of my thoughts on microservices, event sourcing and data access these days :)

Emmanuel

[Emmanuel Bernard]

Hi Nathan and all,

Here are a few random thoughts triggered by your proposal.

== Queries

I find the annotation driven query model relatively hard to read and confusing.

Plus it does not really bring type-safety as fragments of your queries are embedded in annotations as string.

The other problem is that query restrictions are inherently a tree of predicates and you cannot represent a tree of predicate efficiently with annotations. Or at least I haven't found how. You are hitting this problem for example when you add the assumption that all of your predicates will be ANDed, which is a serious limitation.

In decreasing likeliness of query model, I would give the following order (for me):

1. JP-QL or something like that: remains the most compact, expressive and can pretty much write all queries

2. Something like QueryDSL, they made JPA Criteria right (or definitely better)

3. Rails, Grails or Spring Data method name based queries

    Granted you cannot write all queries by any means and thus the scalability in complexity quickly fails

    But at least queries are very readable

4. The annotation based approach that you proposed, which I find the hardest to read

== POJOs

Where do you plan to draw the line? Is your POJO limited to a single object and thus I imagine a single table / non nested structure?

Or are you planning associations / embedded objects etc.

I'm asking because these "details for later" are where models do break apart in my experience.

You seem to want to have the key of the object in the object (otherwise how do you implement `update(Item)` but at the same time you don't want the POJO to have annotation and specifically not have a @Id marking the key. Any reason? I'm not sure how well that can work in practice.

== CDI

I advocated for CDI and its extension model to get something like Spring Data for JPA for a long time. I never had the bandwidth and I wished someone had stepped up. So +1 on my side for that.

== Stateless vs stateful object mapper

In your model (DataAccessTemplate), you imply a stateless model (e.g. the update operate takes the old and new object somehow).

Specifically on this one, I am curious how a "full" app would look like. Having to keep around the old object and create a new updated one seems not so nice and model in practice.

I think at some stage JPA could / should explore a JPA approach but without EntityManager state. You would keep all of JPA but be more tailored to short burst of data update / fire and forget that I think reactive and to some extends microservices have a higher ratio of.

== Update by methods

I found it hard to read what the business method was doing (like queries really).

I see @Decrement and @UPdate, what else do you plan to have as mutator operators?

== Streams

I like the idea of exploring returning Streams for your results. But better have it withing the overall reactive discussion.

== NoSQL, Transactions, configuration etc

You want support for any backend. How do you plan to implement the query model in case the backend is query-less or require quite ad-hoc structures to address queries.

Things like key/value stores come to mind.

Your business code example sort of imply transactions to work well but here you define no transaction demarcation. 

So either we embrace the non transactional nature but we need to offer primitives to work around that, or we need transaction demarcations.

Which abstraction for the non (or loose) transactional nature is the big question.

You are mentioning that you want a no configuration solution (at least no persistence.xml).

You address the entity metadata collection by passing it to your template object as type. But what about database connection, the expected backend (if you can switch between them) and any other configuration level optimisation?

That ended up being a long one and I'm not sure my thoughts are well structured but if I tried to summarize I would say:

- +1 on a CDI based data access / repo like Spring Data

- we could explore stateless JPA as a complement to the CDI template

- we need to nail down the expected transactional model we interact with

- we need to nail down the anticipated limits usage wise of such a solution (associations or not, all backends vs RDBMS etc)

- you should drill down with details on why you see the annotation model for queries / updates as providing an advantage

I know my tone must sound negative but don't be discouraged. I'm hitting ideas but my thoughts fulling expect to be hit back :)

Emmanuel Bernard

Red Hat

[Gunnar Morling]

Nathan, James, all,

Thanks for starting this thread, it's a very important topic for MicroProfile. There are many interesting questions/suggestions in here and I hope to be able to comment on a few more, but one thing specifically caught my attention:

> 2) Complementary event log. In this, you have a traditional CRUD schema,

> but with every update you do to your schema, you also persist an event to 

> the event log in the same transaction. This event log can then be published 

> to a message broker without needing transactions, and updates can be 

> propagated with at least once messaging guarantees to other services.

This sounds very similar to what we do with CDC (change data capturing) in general and the Debezium (http://debezium.io/) project in particular. Only that the event log is in fact the one used maintained by the DB itself and that then Debezium connects to that log and creates change data streams based on table row changes. So the event steam comes "for free" for upstream applications, they don't have to do anything for that other than writing to the DB tables. Also in that approach there's a single source of truth (the DB's log) which is exposed in a (largely) uniform fashion by Debezium.

I think it's a great pattern for pushing data changes between microservices and I'd be very glad to help and propagate it in the context of MicroProfile.

--Gunnar

[Ramesh Reddy]

> - Define an API that works for both relational and NoSQL database

In my experience providing single API over a variety of data sources kind of challenging and limiting. Whatever the query language you choose to define at this API layer must need be translated to native APIs which never map 1 to 1 schema wise, which will force you to interpret the native data in a certain opinionated way where it may lose its original form. Or at a minimum, you will be forced to inject data source specific extensions which will defeat the purpose. These additional layers will also add some amount of overhead. Take look at http://teiid.io where we provide SQL based access to a variety of sources. +1 if there is a way to achieve this.

On Tuesday, June 5, 2018 at 4:24:14 PM UTC-5, Nathan Rauh wrote:

[Nathan Rauh]

Addressing some of the questions that have been asked thus far,

> you cannot represent a tree of predicate efficiently with annotations

Totally agree here.  The examples provided were meant to illustrate 3 different possible approaches that might optionally be combined together.  The simplest and most severely limiting being built-in template methods.  The parameter annotations approach opens up a few additional options (basic mutator operations along with conditions that can only be ANDed together), but as you point out is still very limiting.  The idea for parameter annotations was an attempt to improve upon the usability of method name-based queries which in my opinion rely too much upon non-intuitive side effects.  The method level annotations approach where you specify snippets of JPQL was intended to be the most flexible.  To offer an example of using OR instead of AND, we could switch one of the examples from the original post as follows:

    @Sort("weight")
    @Where("length * width * height > :maxVolume OR weight > :maxWeight")
    Stream<Inventory> findItemsWithExtraShippingCharge(double maxVolume, double maxWeight);

> Where do you plan to draw the line? Is your POJO limited to a single object and thus I imagine a single table / non nested structure? Or are you planning associations / embedded objects etc.

My intent was to get a few ideas out into the community and see where things went from there rather than a complete proposal with every decision made in advance.  My own opinion on this would be to keep it simple (especially for a first version) to cover just the most basic/common patterns, which would also have the effect of making it possible for a wider range of databases to support it.  I could see either disallowing associations/embedded objects entirely, or maybe using something like JSON-B to store any nested objects in columns of the same table.

> You seem to want to have the key of the object in the object (otherwise how do you implement `update(Item)` but at the same time you don't want the POJO to have annotation and specifically not have a @Id marking the key. Any reason? I'm not sure how well that can work in practice.

The intent was actually to allow for either POJOs with or without a key/id.  I want to start with the assumption that you won't necessarily be able to edit all of the POJO types that you need to persist.  For example, some might be classes from a third-party library.  Consequently, I was envisioning a way to have the @Data annotation identify the id field on behalf of the POJO if it has one, which I called 'key' in the example, @Data(key = "productId").  
Note that for the case where the POJO doesn't have an id, not all of the methods would be useful.  You wouldn't use `update(Item)`.  Instead, you would use,

    public boolean insert(K key, T item);

    public boolean updateBy(K key, T updatedItem);

> I see @Decrement and @UPdate, what else do you plan to have as mutator operators?
>

Off the top of my head, here are a few other mutator operators that I could see as parameter annotations:
@Decrement, @Increment, @Assign, @Append
If parameter annotations end up being included, would need to decide if it is worth it to have others like @MultiplyAndAssign, @DivideAndAssign

I was envisioning @Update being the only mutator operator at method level.  Due to its usage of JPQL, it should be highly flexible and able to do much more than the combination of mutator operators at parameter level.

> You want support for any backend. How do you plan to implement the query model in case the backend is query-less or require quite ad-hoc structures to address queries.

The intention wasn't to support every backend out there, but to have a simple enough model that it will be possible to apply in a consistent manner to some of the NoSQL databases as well as relational databases.  Even then, it's certain to end up in a situation where some of the defined operations will not be supportable by some backends.

> You are mentioning that you want a no configuration solution (at least no persistence.xml).

> But what about database connection, the expected backend (if you can switch between them) and any other configuration level optimisation?

Yes, there will need to be configuration.  I chose not to focus on that piece in the initial ideas other than that it should be possible to define configuration entirely within the application.  Requiring a persistence.xml would be a significant obstacle to usability so we should aim to avoid that.



Nathan Rauh
____________________________________________
Software Engineer, WebSphere Application Server
IBM Rochester Bldg 050-2 H215-1
3605 Highway 52N
Rochester, MN 55901-7802

--
You received this message because you are subscribed to a topic in the Google Groups "Eclipse MicroProfile" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/microprofile/1sXozNjXLsM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to microprofile...@googlegroups.com.

To post to this group, send email to

microp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/b3aa33b4-857d-4da6-9ec7-eab396f62096%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Raymond Auge]

Hello all,

I'm interested in this also. I see much interest in having a simple persistence mechanism which has a clean programmatic API and is micro-service oriented.

However, I wonder if the conversation should be flipped on it's head starting with a set goals and from there looking at the best API to represent them.

If one of the goals is to be able to back the API with whatever persistence backend you want, then that severely limits what the API can do _well_. Of course you can make an API that does everything, it just ends up being a nightmare to work with.

Sincerely,

- Ray

.
To unsubscribe from this group and all its topics, send an email to microprofile+unsubscribe@googlegroups.com.

To post to this group, send email to

microp...@googlegroups.com.

To view this discussion on the web visit

https://groups.google.com/d/msgid/microprofile/b3aa33b4-857d-4da6-9ec7-eab396f62096%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "Eclipse MicroProfile" group.

To unsubscribe from this group and stop receiving emails from it, send an email to microprofile+unsubscribe@googlegroups.com.

To post to this group, send email to microp...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/OF7E23D315.A9B265C7-ON002582A4.004BA3E1-862582A4.0054726B%40notes.na.collabserv.com.

For more options, visit https://groups.google.com/d/optout.

--

Raymond Augé (@rotty3000)

Senior Software Architect Liferay, Inc. (@Liferay)

Board Member & EEG Co-Chair, OSGi Alliance (@OSGiAlliance)

[James Roper]

On Wed, 6 Jun 2018 at 17:50, Emmanuel Bernard <emma...@hibernate.org> wrote:

Hey James,

While I am sympathetic to the idea of event sourcing, my experience has been that it is a relatively high bar for teams and I don't think it should be the only model. Event Sourcing is not a bunch of inserts and updates put in a log as you know and of a much higher level business meaning than what Java people call entities.

Also, I've seen and pushed a model where indeed data changes and provided as input to a microservice. But that does not preclude the said microservice to have an actual datastore underneath and thus wanting some more traditional data access technologies.

Anyways, that's a brief version of my thoughts on microservices, event sourcing and data access these days :)

You're absolutely correct that it's a high bar, and that's why I don't think that event sourcing should be the only API offered.

The bigger concern that I have is the one of consistency. Without ACID transactions (which, when you split a system into microservices, you lose because an operation that triggers updates in more than one service cannot be done in a single transaction), you lose consistency. Very few business applications can tolerate this - what business would be happy with a system that lost orders, payments, misreported inventory, etc? So, a persistence solution for microservices I believe must offer a solution for consistency in the place of ACID transactions. Event sourcing is one approach, but as I mentioned there are others - and they generally all centre around an event log of some sort (though not necessarily event sourcing).

And so that's what I think is the most important thing to discuss, there are many very good, well established, mature CRUD APIs out there that we can model a new persistence API off, as long as a MicroProfile specification takes them into consideration, I don't think there's much that can go wrong. But in solving consistency in microservices, that's going into territory that is quite new for most of the industry, and that's where I think a lot of discussion needs to be had.

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/dab72327-e5ec-4a7d-a026-ac5eae3bb08e%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Ken Finnigan]

Could https://github.com/eclipse/microprofile-lra answer some of the concerns around ACID with microservices?

To unsubscribe from this group and stop receiving emails from it, send an email to microprofile+unsubscribe@googlegroups.com.

To post to this group, send email to microp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/dab72327-e5ec-4a7d-a026-ac5eae3bb08e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

James Roper
Senior Developer, Office of the CTO

Lightbend – Build reactive apps!
Twitter: @jroper

--

You received this message because you are subscribed to the Google Groups "Eclipse MicroProfile" group.

To unsubscribe from this group and stop receiving emails from it, send an email to microprofile+unsubscribe@googlegroups.com.

To post to this group, send email to microp...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/CABY0rKNkKdsFyEu_dWPEEfZineOR9CG32Pq%3D%2BJUpV%2BFuFK04zg%40mail.gmail.com.

[James Roper]

On Thu, 7 Jun 2018 at 12:29, Ken Finnigan <k...@kenfinnigan.me> wrote:

Could https://github.com/eclipse/microprofile-lra answer some of the concerns around ACID with microservices?

I don't think so. From my reading of the README, LRA solves problems where you can't do strongly consistent validation at a business level. It looks a lot more like Process Managers, as defined by Microsoft here:

https://docs.microsoft.com/en-us/previous-versions/msp-n-p/jj591569(v%3dpandp.10)

That's about making decisions when you don't have all the information necessary to validate them because that information lives in other services, and then doing compensating actions in the case that those decisions were, further down process, found to be invalid. But to implement this, you still need a level of consistency to get the messages between the services in the first place.

For example, there's a big difference between these two things:

1. An inventory service receiving an event for an order for an item, finding its out of stock, and having to go back to the original order and invalidate it.

2. An inventory service receives an event for an order for an item because at the time that event was published, the inventory service was down, or the network was down, or the service publishing the event crashed before it had an opportunity to publish it, etc.

microprofile-lra appears to be solving the first, but it doesn't address the second. I think a persistence solution must address the second, and microprofile-lra is dependent on the second to work.

Generally, as part of implementing the second, you publish a message to a message broker, and the message broker ensures the message gets there. But how do you publish the message to the message broker atomically with your database update? If you publish during your transaction, and then you crash (or the database crashes, network goes down, etc etc) after that but before the transaction commits, then you get a phantom event going to the inventory service, an event saying this order was submitted when the order service due to the transaction being rolled back believes it was never submitted. If you publish after the transaction, and you crash before the publish but after the commit, then you end up with the message for a submitted order never reaching the inventory service, and so the inventory service never has an opportunity to either fulfill it, or invalidate it with a compensating action as LRA is designed to allow. This is the two generals problem, it's impossible to solve if you try to do it in this way.

However, if you use an event log to publish, then you can solve it. The event log lives in the same database that your data is stored in. When your order service receives a command to submit the order, in addition to updating the order status in the order table, it persists an event to its event log saying the order is submitted. Since the event log and the order table are in the same database, they can be updated in the same ACID transaction, and so you have guaranteed consistency between the event log and the order table. Then, another process, working in the background, polls the event log for events that have not yet been published. This process publishes events to the message broker, and typically by tracking an offset that is associated with each event in the event log, it can guarantee at least once delivery of events from the event log to the message broker. And so, you have end to end consistency. You can even get exactly once delivery if your message broker supports tracking the event log offsets for you.

It sounds like a lot of overhead to implement, and it is if you do it manually. But, if the framework provides in built support for it, then it can be very straight forward. The process that publishes the event log could be built on the MicroProfile Reactive Messaging work that we're doing, in fact, here's an example of doing exactly that in Lagom, the @Incoming (a microprofile messaging annotation) source is the event log, and the @Outgoing destination is Kafka, and all the mechanics of polling, tracking offsets, getting into Kafka etc are handled by the framework, the user only worries about the business logic of deciding which events get published, and converting the internal persisted event representation to an external representation suitable for consumption by other services:

https://github.com/jroper/streaming-new-jakartaee/blob/master/bidding-impl/src/main/java/com/example/auction/bidding/impl/BiddingServiceImpl.java#L90-L92

So, if the persistence API we offer can offer a way to get events into the event log consistently (ie, in the same transaction as the update), then guaranteed consistency in publishing events to other services will be straight forward.

 

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/dab72327-e5ec-4a7d-a026-ac5eae3bb08e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

James Roper
Senior Developer, Office of the CTO

Lightbend – Build reactive apps!
Twitter: @jroper

--
You received this message because you are subscribed to the Google Groups "Eclipse MicroProfile" group.

To unsubscribe from this group and stop receiving emails from it, send an email to microprofile...@googlegroups.com.
To post to this group, send email to microp...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/CABY0rKNkKdsFyEu_dWPEEfZineOR9CG32Pq%3D%2BJUpV%2BFuFK04zg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Eclipse MicroProfile" group.
To unsubscribe from this group and stop receiving emails from it, send an email to microprofile...@googlegroups.com.
To post to this group, send email to microp...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/CAKeeVe5jMfJkP3OdyKXGYRbDmQB3_FTXqeU-7c0-XxT5xPYGGQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[James Roper]

Sorry, just rereading my email, I left out a crucial "never", the second example should be:

2. An inventory service *never* receives an event for an order for an item because at the time that event was published, the inventory service was down, or the network was down, or the service publishing the event crashed before it had an opportunity to publish it, etc.

[James Roper]

On Wed, 6 Jun 2018 at 23:09, 'Gunnar Morling' via Eclipse MicroProfile <microp...@googlegroups.com> wrote:

Nathan, James, all,

Thanks for starting this thread, it's a very important topic for MicroProfile. There are many interesting questions/suggestions in here and I hope to be able to comment on a few more, but one thing specifically caught my attention:

> 2) Complementary event log. In this, you have a traditional CRUD schema,

> but with every update you do to your schema, you also persist an event to 

> the event log in the same transaction. This event log can then be published 

> to a message broker without needing transactions, and updates can be 

> propagated with at least once messaging guarantees to other services.

This sounds very similar to what we do with CDC (change data capturing) in general and the Debezium (http://debezium.io/) project in particular. Only that the event log is in fact the one used maintained by the DB itself and that then Debezium connects to that log and creates change data streams based on table row changes. So the event steam comes "for free" for upstream applications, they don't have to do anything for that other than writing to the DB tables. Also in that approach there's a single source of truth (the DB's log) which is exposed in a (largely) uniform fashion by Debezium.

I think it's a great pattern for pushing data changes between microservices and I'd be very glad to help and propagate it in the context of MicroProfile.

Automatic capturing of updates and publishing them as messages has a lot of strengths, in particular, it's not prone to developer error in that developers can't forget to manually publish an event. But it also has a major drawback - when you capture the insert/update/delete done on the database, you lose the intention of the event. For example, let's say that the status of an order goes to "review". What was the event that triggered that? Did the payment processor reject the payment, and so now someone needs to review it? Did the inventory service say that the order can't be fulfilled, and so now someone needs to review it? Depending on what that intention was, different services that receive the event are going to need to do different processing, for example, an email notification service is going to send emails to different people with different wording based on what the event was, not based on what the data update was. The event is more than just the update, and knowing what the event is is important because other services might need to do different processing based on which event was received.

So capturing data updates requires reverse engineering the updates to work out what the event was, and this is both error prone, and in some circumstances requires a lot of code to get right. In the end, you're going to be converting these updates to some event and passing them to an API to be published, it's much easier to do this when you are handling the command in the first place and know exactly what the intention was and so you can publish the right event. That's why I think an API where you manually publish the event is much simpler for developers to use, and a well written API can offer protections to ensure that they don't forget to emit an event when doing updates, by, for example, requiring to emit an event before an update is committed.

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/29b5c66f-3c78-4727-8072-d12f649120d3%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Nathan Rauh]

I've been trying to give some thought to some ways that event logging might be incorporated into a data access spec.

It seems that it would need a variant of the generic template that allows for supplying an event/reason to the generic insert/update/remove methods, along with some additional built-in methods for querying the event log.

Also, an @Event annotation that could be tacked on to a method parameter with which the application could identify the reason for the update operation.  For example,

@Update("quantity = quantity - :amount")
@Where("productId = :productId AND quantity >= :amount")
boolean decrementQuantity(@Event DecQuantityReason reason, String productId, long amount);

To run through an example here, assuming that InventoryItem "PRD000203" has quantity of 100, then the following query,

inventorydb.decrementQuantity(DecQuantityReason.REMOVED_FOR_SHIPMENT, "PRD000203", 5);

This would result in 3 commands to the database, all within the same transaction, in the following order,
- the update made to the InventoryItem table as usual
- a query on the InventoryItem table to retrieve the newly computed quantity value, which in this case is 95
- an insert to the InventoryItemEvents table, of something along the lines of:
    sequenceNumber, timestamp, "REMOVED_FOR_SHIPMENT", "PRD000203", "{ quantity: 95 }"

Having stepped through this, it seems like a lot of overhead to put in place in advance of the comprehensive solution across MicroProfile technologies.  One option here would be to design a spec that keeps event logging in mind, remaining consistent with being able to add support as work progress across the MicroProfile platform, but not covering the event logging support in the initial release.  Any thoughts on whether that would be acceptable?

From:        James Roper <ja...@lightbend.com>
To:        MicroProfile <microp...@googlegroups.com>
Date:        06/06/2018 10:01 PM
Subject:        Re: [microprofile] Re: Data access spec for MicroProfile
Sent by:        microp...@googlegroups.com

---

--
You received this message because you are subscribed to a topic in the Google Groups "Eclipse MicroProfile" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/microprofile/1sXozNjXLsM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to microprofile...@googlegroups.com.

To post to this group, send email to

microp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/CABY0rKO6M3tvQ5F2P2mmt2Rk3fFqOPXPB%3DdW5TrByytvcOK6gA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Craig McClanahan]

A lot of these ideas make sense, but a couple things puzzled me.  Embedded below.

On Thu, Jun 7, 2018 at 2:28 PM Nathan Rauh <natha...@us.ibm.com> wrote:

I've been trying to give some thought to some ways that event logging might be incorporated into a data access spec.

It seems that it would need a variant of the generic template that allows for supplying an event/reason to the generic insert/update/remove methods, along with some additional built-in methods for querying the event log.

Also, an @Event annotation that could be tacked on to a method parameter with which the application could identify the reason for the update operation.  For example,

@Update("quantity = quantity - :amount")
@Where("productId = :productId AND quantity >= :amount")
boolean decrementQuantity(@Event DecQuantityReason reason, String productId, long amount);

To run through an example here, assuming that InventoryItem "PRD000203" has quantity of 100, then the following query,

inventorydb.decrementQuantity(DecQuantityReason.REMOVED_FOR_SHIPMENT, "PRD000203", 5);

That part makes sense -this the delta change being recorded.

This would result in 3 commands to the database, all within the same transaction, in the following order,
- the update made to the InventoryItem table as usual
- a query on the InventoryItem table to retrieve the newly computed quantity value, which in this case is 95

Why is this necessary?  Probably related to my next comment.

- an insert to the InventoryItemEvents table, of something along the lines of:
    sequenceNumber, timestamp, "REMOVED_FOR_SHIPMENT", "PRD000203", "{ quantity: 95 }"

This is where I am confused.  If I am just reading the events table, I am likely to assume that 95 items just got removed for shipment.  Shouldn’t this really be a quantity of 5?

Separately, I assume you can change multiple values in one event, right?  Say the shipping address was also changed.  If so, we would want a way to distinguish increment/decrement operations from replace operations, which might be why you picked the replacement value of 95.  Makes it hard to see how you could replay events to rebuild after a problem though.

Having stepped through this, it seems like a lot of overhead to put in place in advance of the comprehensive solution across MicroProfile technologies.  One option here would be to design a spec that keeps event logging in mind, remaining consistent with being able to add support as work progress across the MicroProfile platform, but not covering the event logging support in the initial release.  Any thoughts on whether that would be acceptable?

Craig McClanahan

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/OF1C23D9BB.E9756BB9-ON002582A5.004E8D7E-862582A5.0075F803%40notes.na.collabserv.com.

[Mark Swatosh]

Hi Craig,

I'll try to answer your questions.

Why is this necessary?  Probably related to my next comment.

This is where I am confused.  If I am just reading the events table, I am likely to assume that 95 items just got removed for shipment.  Shouldn’t this really be a quantity of 5?

What doesn't seem to be clear here is that the event table would be containing new values which have changed from the previous state (represented by the sum of the previous events). So in the example, 95 is the new value of 'quantity'.  "REMOVED_FOR_SHIPMENT" is the logged reason for the event, though that could probably be improved by passing in the number removed so it could be clearer e.g. "5 items removed for shipment"

Separately, I assume you can change multiple values in one event, right?

I think it would be important to support that, so if you're rebuilding from the event log, and you have a case where 5 items were removed from one place and added to another, you can't end up in a state with 5 additional or missing items.

That said, there may be better ways to do this. It sounds like James is knowledgeable in this area, so I'm interested in what he has to say.

Thanks,

Mark 

[Craig McClanahan]

On Thu, Jun 7, 2018 at 7:07 PM Mark Swatosh <mark.s...@gmail.com> wrote:

Hi Craig,

I'll try to answer your questions.

Why is this necessary?  Probably related to my next comment.

This is where I am confused.  If I am just reading the events table, I am likely to assume that 95 items just got removed for shipment.  Shouldn’t this really be a quantity of 5?

What doesn't seem to be clear here is that the event table would be containing new values which have changed from the previous state (represented by the sum of the previous events). So in the example, 95 is the new value of 'quantity'.  "REMOVED_FOR_SHIPMENT" is the logged reason for the event, though that could probably be improved by passing in the number removed so it could be clearer e.g. "5 items removed for shipment"

So what happens if there are a -5 and a -10 transaction you have to both revoke from?  Neither 95 or 90 is the right answer if you are taking absolute values.  And you are pot luck depending on the order you replay events.

Ah the fun of micro services.  😀

Craig

Separately, I assume you can change multiple values in one event, right?

I think it would be important to support that, so if you're rebuilding from the event log, and you have a case where 5 items were removed from one place and added to another, you can't end up in a state with 5 additional or missing items.

That said, there may be better ways to do this. It sounds like James is knowledgeable in this area, so I'm interested in what he has to say.

Thanks,

Mark 

--

You received this message because you are subscribed to the Google Groups "Eclipse MicroProfile" group.
To unsubscribe from this group and stop receiving emails from it, send an email to microprofile...@googlegroups.com.
To post to this group, send email to microp...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/0ce579f9-9868-4c43-92b5-14484e3ded1d%40googlegroups.com.

[James Roper]

Hi all,

A few thoughts on my experiences in this area.

My primary experience on event logs has been with event sourcing proper, Lagom's persistence API is event sourced based. Note that event sourcing does not exclude the need for CRUD APIs for database access, but generally, the CRUD APIs are used in projections, or read side processors, that take events and update views specific for the queries that need to be made of them. What we've found, and this has already been mentioned in this thread, for many developers, moving from a traditional approach to developing systems to event sourcing is too big a leap, especially if the move is coupled with moving to microservices, moving to asynchronous messaging, moving to asynchronous io, moving to NoSQL, etc etc etc.

So, we've put a fair bit of effort into rethinking this, and asking is there a middle ground. The underlying goal that we don't want to compromise is ensuring that consistency is maintained - ensuring that, without distributed transactions, you can do a database update and publish to a message broker, effectively atomically (effectively because all the solutions we've come up with are eventually consistent, so they are eventually atomic). But, to be completely honest, although we have prototyped some APIs that we are fairly happy with, none of what we've implemented has yet made a production release, so my experience here is of limited (though not zero) value, we don't have feedback from actual users building actual systems, and how difficult or not they find it to use these features.

From my experience in prototyping these solutions, here's what I've found.

If the datastore supports ACID transactions (and for our use case, which is not necessarily the same as Microprofiles, we made the decision that we're only going to consider datastores that support ACID transactions, if a user is game enough to switch to NoSQL, then they're probably game enough to use event sourcing and so don't need these features), then we can provide an API that feels just like what developers actually want to do - imperatively do a CRUD update, and then push an event to a message broker, and have those atomic.

So, imagine doing this without any consistency safety, you might have a message broker component, and a DAO that you're using, and you do this:

@Transactional

public void doSomeOperation(SomeCommand command) {
  dao.doSomeUpdate(command);

  CommandCompleted event = new CommandCompleted(command);

  messageBroker.publish(event);

}

Of course, the above is not atomic (unless you're using distributed transactions, but we want to avoid them because most cloud messaging technologies don't support distributed transactions).

So, what if we can provide an event log in the place of the message broker component, that instead of pushing to the message broker, it persists to the event log, for publishing later (which we already have implemented, and the MicroProfile messaging spec we're working on will support too)? And that's exactly what we did. So the code simply becomes:

@Transactional

public void doSomeOperation(SomeCommand command) {
  dao.doSomeUpdate(command);

  CommandCompleted event = new CommandCompleted(command);

  eventLog.publish(event);

}

In this case, the event gets serialized and stored in the database serialized, the same way it would be serialized if being sent to a message broker. The nice thing about a solution like this is that it's such a small step from a code perspective, and really, I don't think it's that big a step conceptually, it's just a matter of having an intermediate place that the event is placed before going to the message broker. So it's easy to document and describe to developers. The problem with the above solution is that it's easy to forget to emit an event, and it's also easy to emit events that don't fully or correctly capture the update. But that's where we're up to, we don't yet have any prototypes or concrete ideas for APIs that try and force you to emit an event, for example. I'm not sure whether such an API would work against the goal of having something that is conceptually a small step for developers, that would depend on what the API looked like.

Also, one final consideration, a number of database drivers out there are starting to support asynchronous IO, and Oracle is hard at work at the Asynchronous DataBase Adatper (ADBA) spec, essentially an asynchronous JDBC. It would be good if such an API was able to support asynchronous operations. The hard thing about asynchronous operations in the context of persistence is transaction context - propagating the thread local context necessary to do this is very difficult to do, and often puts a heavy burden on the developer. In my experience, the best way to support this is to allow explicit passing of the transaction context (in addition to implicit passing via thread locals).

Cheers,

James

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/CANgkmLAPA3JNVBV0ZATok7stuQ6cnnU6REgfs1Pj3RpL6OtDyA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Mark Swatosh]

So what happens if there are a -5 and a -10 transaction you have to both revoke from?  Neither 95 or 90 is the right answer if you are taking absolute values.  And you are pot luck depending on the order you replay events.

My thought would be this could go three ways

100 > 95 > 85 (-5 then -10)

100 > 90 > 85 (-10 then -5)

100 > 85 (-10 and -5)

So then as long as you know what you've revoked, you would end up in an acceptable state? Maybe I'm missing something here... :-)

From James's reply, I'm leaning toward keeping the first version of this simple and focused on CRUD operations, and adding Event Logging/Sourcing in a future version, potentially after the messaging spec is complete?

Thanks,

Mark

[Craig McClanahan]

In Nathan's pseudocode example:

>To run through an example here, assuming that InventoryItem "PRD000203" has quantity of 100, then the following query,
>
>inventorydb.decrementQuantity(DecQuantityReason.REMOVED_FOR_SHIPMENT, "PRD000203", 5); 
>

>This would result in 3 commands to the database, all within the same transaction, in the following order,
>- the update made to the InventoryItem table as usual
>- a query on the InventoryItem table to retrieve the newly computed quantity value, which in this case is 95

>- an insert to the InventoryItemEvents table, of something along the lines of:
>    sequenceNumber, timestamp, "REMOVED_FOR_SHIPMENT", "PRD000203", "{ quantity: 95 }"

The incoming event says to decrement by five.  But the proposed insert into InventoryItemEvents table stored the quantity as 95 (i.e. the initial 100 minus the five that were removed.  That's where I was getting confused - I would have expected the quantity in the stored event to be 5, which would mentally match the semantics of "REMOVED_FOR_SHIPMENT" as the event type in my mind, as well as making replays or undoes more understandable.

Craig

--

You received this message because you are subscribed to the Google Groups "Eclipse MicroProfile" group.
To unsubscribe from this group and stop receiving emails from it, send an email to microprofile...@googlegroups.com.
To post to this group, send email to microp...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/704201fe-4f9c-499e-ba3c-ef93f4185f57%40googlegroups.com.

[Nathan Rauh]

Having a set of goals is a good idea.  There were a few partial goals in the initial post, although overshadowed by the example ideas, and some others have been scattered across the various responses.  I went through all of the responses again and have tried to collect a good list.  I'm sure it will reflect my own biases, but should at least open up for further discussion.

Goals:

High Usability
- Standardize and greatly simplify basic patterns for data access. Spec is not tied to any particular data access technology.
- Avoid the need for the user to write boilerplate code, and the need to manually manage life cycles of objects. The most basic of CRUD operations should be possible in just a couple lines of code (Inject, then run)
- Allow the user to think and operate purely in terms of their own data model, but without extra steps such as writing or building additional classes to model their data.  It must be possible to use existing POJOs as is, without modification. Existing POJOs might or might not include attributes such as a primary key/id. A good spec should be able to cope with both.
- Near zero learning curve. Must be obvious to first-time users given only JavaDoc of the top level annotation or class and standard auto-complete. Intuitive, with no magic side-effects such as method names controlling details of updates/queries.

Limited in nature, especially so for initial release
- Extensible enough for future addition of event logging.
- Covers a core set of most commonly used data access operations. Is not a full replacement for SQL, JPA, or NoSQL vendor APIs.  Covers operations on a single entity type, not across multiple.
- Not overly limited so as to not be useful. Should not restrict to just a single sequence of conditions ANDed together. Must also support OR, nesting of conditions/computations, and so forth as business logic commonly requires, subject to what the the backend database is capable of.
- Ability to run multiple operations within a single transaction if the backend database supports it.

Easy to configure
- Configurable entirely within the application, in a standard way, possibly via MP config.  It would be too high of a learning curve to require a JPA persistence.xml.

.
To unsubscribe from this group and all its topics, send an email to microprofile...@googlegroups.com.

To post to this group, send email to

microp...@googlegroups.com.

To view this discussion on the web visit

https://groups.google.com/d/msgid/microprofile/b3aa33b4-857d-4da6-9ec7-eab396f62096%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Eclipse MicroProfile" group.

To unsubscribe from this group and stop receiving emails from it, send an email to microprofile...@googlegroups.com.

To post to this group, send email to

microp...@googlegroups.com.

To view this discussion on the web visit

https://groups.google.com/d/msgid/microprofile/OF7E23D315.A9B265C7-ON002582A4.004BA3E1-862582A4.0054726B%40notes.na.collabserv.com.
For more options, visit https://groups.google.com/d/optout.

--
Raymond Augé (@rotty3000)
Senior Software Architect Liferay, Inc. (@Liferay)
Board Member & EEG Co-Chair, OSGi Alliance (@OSGiAlliance)

--

You received this message because you are subscribed to a topic in the Google Groups "Eclipse MicroProfile" group.
To unsubscribe from this topic, visit

https://groups.google.com/d/topic/microprofile/1sXozNjXLsM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to microprofile...@googlegroups.com.

To post to this group, send email to

microp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/CAMm6HcBXsYdrQbOyMxMKv-JUZVBUbJSrrYS1mdQx6KqtQaXPWA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Reza Rahman]

I have recently switched jobs and now work for AxonIQ, the folks that created the Axon framework. A large part of what Axon does is help applications implement Event Sourcing and CQRS. We see things like event sourcing and technologies like JPA (and NoSQL) to be highly complementary. Indeed we have JPA support baked into Axon. Many of our users use JPA to implement materialized views from event sourced data, in both monoliths and microservices. We do indeed recommend event sourcing and CQRS as good ways to synchronize distributed data.

Doing something with event sourcing and CQRS in MicroProfile might be worth considering. I can even ask my employer if they would be willing to help and perhaps make Axon a compliant implementation.

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/CABY0rKOEKR3Ey71iZ92GHj66HAJn0gKxUd4K-Rn3rnHsxfM2wA%40mail.gmail.com.

[Edwin Derks]

Good luck with your job @ AxonIQ, Reza!

Me and my team are using Axon in my current project, and building a new platform with the latest Axon capabilities. AxonIQ's is involved in getting our new platform started, so we have short ties to AxonIQ.

The reason I'm mentioning this, is the fact that Axon is new built on Spring in a lot of area's. I discussed this with my team because I was wondering if we could built our new platform with Axon (which we want to use), but not necessarily Spring. As far as we could determine, Axon doesn't work on a stack other than Spring, at least not out-of-the-box.

As far as my knowledge goes, we need Spring for Axon, but is that really necessary? I mean, Axon leverages CQRS and EventSourcing in it's framework but is it really necessary to bootstrap all that with Spring? IMHO I think that it doesn't, and if we can define an Axon specification (microprofile-axon?, microprofile-cqrs?) that defines the Axon specification, it can be implemented on Java EE/Jakarta EE/Microprofile stacks. This would give Axon the power to move outside of the Spring environment and apply to Java EE environment as well, making it even a more powerful and widely used framework!

Next to that, with all the cloud development around, my team is busy integrating the current Axon/Spring stack as much as possible on cloud (AWS) components, leveraging the data persistence, scalability, performance and monitoring of the cloud as much as possible. If we have a specification for Axon, we could even build an implementation based on Microprofile, optimized for the cloud. That is where I see so much potential in having a specification coupled loose from it's implementation, unlike the current Spring/Axon lock-in that is now available.

I would like to know how you guys feel about this. Do you agree with me? Is my knowledge of Spring/Axon accurate in this context? If there really is going to be a specification for Axon in Microprofile, i certainly would like to contribute!

Kind regards,

Edwin

[Nathan Rauh]

Craig,

I'm sorry to have used such a poor example.  As you pointed out, an event that records subtraction of a numeric value can be optimized.  A better example might be removing a percentage of the 'quantity' as determined by a separate 'ratio' column, which might have changed by the time the revert is being requested.  I don't think timestamps/sequence numbers on the event log updates would be sufficient (the CRUD operations to which they pertain might not match that order exactly) when crossing multiple columns to ensure values are precisely replayed from the log.  In order to get sufficient information into the event log there would need to be some fetching of current values that wouldn't otherwise be required by the actual CRUD operation.  Possibly someone will point out that this sort of update shouldn't be allowed when using event logging, and if so it might be preferable to prevent that at the level of the API.  Regarding why logging only a current value can be sufficient, it's because the log can be queried for the prior update as well as the one being reverted, and the correct difference computed between them.  I don't want to dwell on the event log detail too much here.  I'd like to defer these sort of discussions until later and get a basic framework off the ground first that event logging can subsequently be added onto.  The point I was trying to raise is that there is an amount of overhead and complexity here that we only want to incur when event logging is specifically asked for.

From:        Craig McClanahan <crai...@gmail.com>
To:        microp...@googlegroups.com
Date:        06/08/2018 03:46 PM
Subject:        Re: [microprofile] Re: Data access spec for MicroProfile
Sent by:        microp...@googlegroups.com

---

--
You received this message because you are subscribed to a topic in the Google Groups "Eclipse MicroProfile" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/microprofile/1sXozNjXLsM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to microprofile...@googlegroups.com.

To post to this group, send email to

microp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/CANgkmLC0wkSapoffEUz9JTcxurbJ9tYQZWMvRVMo5bNepZwkyA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[James Roper]

Hi Reza,

That sounds fantastic. I would love to work with you or anyone else at AxonIQ to help produce an Event Sourcing/CQRS spec for MicroProfile. We would also seek to make an Akka persistence/Lagom compliant implementation.

Cheers,

James

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/2c7c886d-5bfc-22b2-9615-d9cdae86c30b%40gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Galder Zamarreno]

Hi,

My name is Galder Zamarreño, I'm part of the Infinispan team at Red Hat and I was part of the JSR-107 expert group.

I feel that the API has sync access and local in-memory access in mind, and this is a limited subset of the possible implementations.

For example, it doesn't seem to take into account clustered implementations nor remote implementations. A clustered and/or remote implementation will go over the network and the API is sync. Get ready to block...

APIs that rely on contains/comparison based on value could be particularly slow in remote implementations when the value to compare with will always be sent over the network. If the values are big that could be a problem. 

IMO, the proposed API does not add much that JSR-107's JCache already has in it. 

Cheers

[Nathan Rauh]

Discussions are headed in a different direction, but I thought I'd offer one additional thought on the traditional CRUD approach that might allow a partly type safe and fluent query builder API with no pre-compilation.  If the template provided handles to the various attributes, for example,

quantity = inventorydb.numeric("quantity");

it could allow for generic template find/update/remove methods such as,

    public Stream<T> find(Condition query, Sort... sorts);
    public T find(K key, Condition query);
    public long update(Condition conditions, Assignment... updates);
    public boolean update(K key, Condition conditions, Assignment... updates);

Example usage corresponding to previous examples,

inventorydb.update(prodId, quantity.greaterOrEq(amount), quantity.decrement(amount));

includesPackaging = inventorydb.bool("includesPackaging");
length = inventorydb.numeric("length");
width  = inventorydb.numeric("width");
height = inventorydb.numeric("height");
weight = inventorydb.numeric("weight");

itemsWithExtraShippingCharge = inventorydb.find(
    length.mult(width).mult(height).greaterOrEqual(maxVolume).or(weight.greaterThan(maxWeight)),
    weight.sortAscending()
);

inventorydb.update(
    includesPackaging.eq(false),
    includesPackaging.assign(true),
    weight.increment(length.mult(width).plus(width.mult(height)).plus(height.mult(length)).mult(2 * w)
);

.
To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/OF7E23D315.A9B265C7-ON002582A4.004BA3E1-862582A4.0054726B%40notes.na.collabserv.com.
For more options, visit https://groups.google.com/d/optout.

[Emmanuel Bernard]

I agree with James, LRA are fundamentally business commits / rollbacks. Whereas how to deal with data consistency is much closer to a technical concern. Since data is not business, it's data, different consistency models are better suited for such a concept.

Of course there is a grey area between a business concern and a technical concern because which consistency you will select will be influenced by your business requirements, but that is I think a separate topic.

Emmanuel

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/dab72327-e5ec-4a7d-a026-ac5eae3bb08e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

James Roper
Senior Developer, Office of the CTO

Lightbend – Build reactive apps!
Twitter: @jroper

--
You received this message because you are subscribed to the Google Groups "Eclipse MicroProfile" group.

To unsubscribe from this group and stop receiving emails from it, send an email to microprofile...@googlegroups.com.
To post to this group, send email to microp...@googlegroups.com.

[Emmanuel Bernard]

Thanks for summarizing the goals here.

While we pursue the idea of a brand new spec and impl, I would like us to also explore the elephant in the room here and try and find a way to evolve our main Java existing data access standard JPA to meet these goals. So for this exercise, let's assume that we can update JPA and/or add new specs built on top of JPA (a not so improbable future).

See inline for some ideas.

Le samedi 9 juin 2018 00:01:11 UTC+2, Nathan Rauh a écrit :

Having a set of goals is a good idea.  There were a few partial goals in the initial post, although overshadowed by the example ideas, and some others have been scattered across the various responses.  I went through all of the responses again and have tried to collect a good list.  I'm sure it will reflect my own biases, but should at least open up for further discussion.

Goals:

High Usability
- Standardize and greatly simplify basic patterns for data access. Spec is not tied to any particular data access technology.

Adding to JPA something akin to the Spring Data templating model might help towards the simplification goal.

While JPA has been designed with RDBMS in mind, our efforts via Hibernate OGM - https://hibernate.org/ogm/ - is showing that you can make it a multi datastore technology if you accept the fact that the implementation will be prescriptive towards the mapping model between object and X.

 

- Avoid the need for the user to write boilerplate code, and the need to manually manage life cycles of objects. The most basic of CRUD operations should be possible in just a couple lines of code (Inject, then run)

Here there are several paths to explore:

- the Spring Data like template as I mentioned above

- offer a stateless EntityManager API if I correctly understand your point about "manage life cycles of objects" (though I would prefer if you clarified that goal).

 

- Allow the user to think and operate purely in terms of their own data model, but without extra steps such as writing or building additional classes to model their data.  It must be possible to use existing POJOs as is, without modification. Existing POJOs might or might not include attributes such as a primary key/id. A good spec should be able to cope with both.

I would love to see a bit more evidence that mapping existing unrelated objects are a big need. I know it would really please the DDD anti-anemic POJO crowd for sure.

On the JPA front, you can map your metadata via XML (well ok, we are in 2018) and we can definitely explore fluent APIs to attach the mapping metadata to a untainted POJO. We have explored things like that for Hibernate Validator (Bean Validation) https://docs.jboss.org/hibernate/stable/validator/reference/en-US/html_single/#section-programmatic-api and more recently for Hibernate Search. These two dance around the idea of metadata on POJOs via a programmatic API so I think it is doable for JPA (even though it is more complex).

As for the id not being in the POJO, we can accommodate JPA and its APIs for that, it does not fundamentally change the JPA concepts.

We can also envision a simpler way to bootstrap JPA without persistence.xml esp if we have the programmatic mapping API.

 

- Near zero learning curve. Must be obvious to first-time users given only JavaDoc of the top level annotation or class and standard auto-complete. Intuitive, with no magic side-effects such as method names controlling details of updates/queries.

Here your mileage may vary. As I explained it my initial email, I find the method name more intuitive than the annotation version so far.

JPA might not it the mark of zero learning curve but I would argue a few things in its favor:

- if you just want persist/read objects (no associations, polymorphism etc), then things are simple

- the stateless EntityManager idea would bring more straightforwardness to the API usage (explicit object action vs object lifecycle managed by the framework)

- JPA is massively used so people know it - according to some definition of know ;p 

 

Limited in nature, especially so for initial release

One good thing about JPA or a JPA-lite version is that we know it could grow to more complex cases: it does today already :)

 

- Extensible enough for future addition of event logging.

Reading James proposal of storing the event log in the same Tx as the CRUD operations, I realised that Hibernate ORM (via the envers module sorta kinda does that already http://hibernate.org/orm/envers/

If you forget about the historical query capabilities of Envers, what it does is for each transaction optionally store a companion entity for which we advertised the idea of attaching the user doing the operation and other cross cutting fields. But we could imagine this insertion as the event log. All that to say that we could retrofit such feature in JPA if we have the will.

 

- Covers a core set of most commonly used data access operations. Is not a full replacement for SQL, JPA, or NoSQL vendor APIs.  

See above.

 

Covers operations on a single entity type, not across multiple.
- Not overly limited so as to not be useful. Should not restrict to just a single sequence of conditions ANDed together. Must also support OR, nesting of conditions/computations, and so forth as business logic commonly requires, subject to what the the backend database is capable of.

The template + stateless API + programmatic mapping API + simpler bootstrap API

 

- Ability to run multiple operations within a single transaction if the backend database supports it.

Checked :)

 

Easy to configure
- Configurable entirely within the application, in a standard way, possibly via MP config.  It would be too high of a learning curve to require a JPA persistence.xml.

The two main reasons to have persistence.xml are:

- if which JAR to scan for @Entity but a programmatic mapping API or explicit entity listing would eliminate that need

- define the data source to access: if we have a simpler way to do that (e.g. via MP config) then that does not sound too hard.

Should collect that answer in a wiki page somewhere?

Emmanuel Bernard

Red Hat

[Reza Rahman]

I think it's definitely useful to start a conversation. More details on the dedicated thread on CQRS/ES: https://groups.google.com/forum/#!topic/microprofile/pbYf_7sVKKA.

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/CABY0rKOnvu34xJg%3D-XpUNeZ3ifgFPEMxNPCcU1krb5LvJon-kA%40mail.gmail.com.

[Reza Rahman]

Edwin,

I am looping in our CTO, Allard Buijze, into this conversation to properly and most accurately depict Axon alignment with Spring, Java EE and CDI. I am dropping out the MicroProfile alias via BCC on this message as at least part of this discussion may be somewhat tangential to MicroProfile. The most MicroProfile centric parts Allard can certainly pick up on the other thread I think.

Of course I'll remain engaged as well so please don't hesitate to reach out to me if needed.

Thanks,
Reza

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/45538a41-57b5-4ba0-b39b-a7521761d8df%40googlegroups.com.

[Reza Rahman]

I am definitively in favor of just moving forward JPA itself with these and other features. My view is that JPA evolution has needlessly slowed down in the past few years. One thing I would like to see for sure is omitting the persistence.xml in Java EE environments whenever possible. I had a JIRA open for this on the former Java EE specification: https://github.com/javaee/javaee-spec/issues/41.

--
You received this message because you are subscribed to the Google Groups "Eclipse MicroProfile" group.
To unsubscribe from this group and stop receiving emails from it, send an email to microprofile...@googlegroups.com.
To post to this group, send email to microp...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/f2dd64e2-db92-44ae-a74e-8380d09fc6b2%40googlegroups.com.

[Werner Keil]

There's JNoSQL and JPA, are you saying there should be another wrapper in MicroProfile?

[Andy Guibert]

Essentially yes.  The original proposal was an abstraction layer above technologies such as JNoSQL and JPA. Frameworks like Spring JPA have made it clear that there is still room for an even higher level of data access abstraction beyond what JPA offers.  However, this proposal does not suggest an API as comprehensive as JPA/JNoSQL.  I think this proposal has value because it covers the easy/medium data access cases, and could allow the user to "unwrap" down to the lower data access layer for the advanced cases.

I agree that this proposal would be suitable for JPA directly, but given the limbo state of JavaEE/JakartaEE I don't see evolving JPA as a possibility now or in the near future.

To recap again, the value-adds on top of JPA would be:

 - Built-in template for most basic CRUD operations

 - User-defined interfaces (think Spring JPA and MP Rest Client do) for easy/medium complexity queries that integrate w/ CDI

 - Simplified configuration model, namely, eliminate persistence.xml and replace with POJO and/or MP Config

Regarding the discussion about CQRS/ES, I think that is certainly an interesting topic and it fits nicely into the microservice architecture, but should be considered separate from what is being proposed here since such a spec could build on top of any traditional data access layer.

- Andy

IBM

[Mark Little]

Worth noting that all transaction protocols, whether ACID or extended such as LRA, assume the system moves from one consistent state to another (the C in ACID is not something that is enforced by the transaction manager, for example).

Haven't had a chance to track this thread much recently but is anyone here working with the MP transaction team? Whether leveraging JPA or coming up with new spec we should make sure these efforts are aligned where it makes sense.

Sent from my iPhone

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/79abc475-a880-44df-9b1a-82c0b2b6a1c6%40googlegroups.com.

[Emily Jiang]

+1 Mark!

I have not fully digested this.  My apologies in advance if I misunderstood the effort.


I see it is important to define a simple data access for the backend database such as MongoDB, etc. This week at EclipseCon France,  Rafael Del Nero presented jNoSQL, which is quite simple to use and it has support on most of backend databases. If it is the technology we need, why do we need to reinvent the whole annotation again. Can we just adopt jNoSQL?

Emily

[Michael Musgrove]

LRA is protocol that guarantees atomicity. Updates made by the various parties involved in the LRA are not isolated from each other but the protocol does guarantee the eventual consistency of the overall business interaction for well behaved participants.

The protocol coordinator achieves this level of guarantee by ensuring that protocol state changes are made durable throughout the interaction. On receipt of a request a participant first registers itself with the coordinator before performing any business updates. The consequence is that in failure scenarios the (reliable) coordinator will let the participant know what the expected state of its business changes should be and the participant can then act to ensure that the expected state is arrived at. Under error conditions the protocol will keep retrying until an end state is reached.

With regards to the reliability of communications, we leave that responsibility to other MicroProfile specifications, suffice to say that due to the reliable nature of the coordinator, the system will eventually reach a consistent state.

On Thursday, 7 June 2018 03:58:28 UTC+1, James Roper wrote:

On Thu, 7 Jun 2018 at 12:29, Ken Finnigan <k...@kenfinnigan.me> wrote:

Could https://github.com/eclipse/microprofile-lra answer some of the concerns around ACID with microservices?

I don't think so. From my reading of the README, LRA solves problems where you can't do strongly consistent validation at a business level. It looks a lot more like Process Managers, as defined by Microsoft here:

https://docs.microsoft.com/en-us/previous-versions/msp-n-p/jj591569(v%3dpandp.10)

That's about making decisions when you don't have all the information necessary to validate them because that information lives in other services, and then doing compensating actions in the case that those decisions were, further down process, found to be invalid. But to implement this, you still need a level of consistency to get the messages between the services in the first place.

For example, there's a big difference between these two things:

1. An inventory service receiving an event for an order for an item, finding its out of stock, and having to go back to the original order and invalidate it.

2. An inventory service receives an event for an order for an item because at the time that event was published, the inventory service was down, or the network was down, or the service publishing the event crashed before it had an opportunity to publish it, etc.

microprofile-lra appears to be solving the first, but it doesn't address the second. I think a persistence solution must address the second, and microprofile-lra is dependent on the second to work.

Generally, as part of implementing the second, you publish a message to a message broker, and the message broker ensures the message gets there. But how do you publish the message to the message broker atomically with your database update? If you publish during your transaction, and then you crash (or the database crashes, network goes down, etc etc) after that but before the transaction commits, then you get a phantom event going to the inventory service, an event saying this order was submitted when the order service due to the transaction being rolled back believes it was never submitted. If you publish after the transaction, and you crash before the publish but after the commit, then you end up with the message for a submitted order never reaching the inventory service, and so the inventory service never has an opportunity to either fulfill it, or invalidate it with a compensating action as LRA is designed to allow. This is the two generals problem, it's impossible to solve if you try to do it in this way.

However, if you use an event log to publish, then you can solve it. The event log lives in the same database that your data is stored in. When your order service receives a command to submit the order, in addition to updating the order status in the order table, it persists an event to its event log saying the order is submitted. Since the event log and the order table are in the same database, they can be updated in the same ACID transaction, and so you have guaranteed consistency between the event log and the order table. Then, another process, working in the background, polls the event log for events that have not yet been published. This process publishes events to the message broker, and typically by tracking an offset that is associated with each event in the event log, it can guarantee at least once delivery of events from the event log to the message broker. And so, you have end to end consistency. You can even get exactly once delivery if your message broker supports tracking the event log offsets for you.

It sounds like a lot of overhead to implement, and it is if you do it manually. But, if the framework provides in built support for it, then it can be very straight forward. The process that publishes the event log could be built on the MicroProfile Reactive Messaging work that we're doing, in fact, here's an example of doing exactly that in Lagom, the @Incoming (a microprofile messaging annotation) source is the event log, and the @Outgoing destination is Kafka, and all the mechanics of polling, tracking offsets, getting into Kafka etc are handled by the framework, the user only worries about the business logic of deciding which events get published, and converting the internal persisted event representation to an external representation suitable for consumption by other services:

https://github.com/jroper/streaming-new-jakartaee/blob/master/bidding-impl/src/main/java/com/example/auction/bidding/impl/BiddingServiceImpl.java#L90-L92

So, if the persistence API we offer can offer a way to get events into the event log consistently (ie, in the same transaction as the update), then guaranteed consistency in publishing events to other services will be straight forward.

 

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/CAKeeVe5jMfJkP3OdyKXGYRbDmQB3_FTXqeU-7c0-XxT5xPYGGQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[James Roper]

Are there any papers out there on this protocol? Any implementations out there? I'd like to read more.

Have you considered CQRS and how it's able to implicitly implement eventually consistent long running actions? My concern is that LRA is just a specialisation of CQRS. CQRS solves this by the way by intrinsically implementing retries of the next action in the process. All that's needed on top is an entity that manages the state of the process and emits state changes as events via the CQRS event log, these state changes are then picked up by CQRS listeners which attempt the next action. This is a well documented approach to long running actions (see https://docs.axonframework.org/part-ii-domain-logic/sagas and http://msdn.microsoft.com/en-us/library/jj591569.aspx), with a number of real world implementations out there in popular use today.

To me it would make sense to create a CQRS spec first, and then layer an LRA/Saga/Process Manager specification on top of that, in the same way that Axon framework does.

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/d29a5526-ce44-4970-a266-624803194252%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Andy Guibert]

Emily,

Yes the original proposal took into account the existence of jNoSQL, and including it in MP has already been proposed and discussed here:

https://groups.google.com/forum/#!searchin/microprofile/jnosql%7Csort:date/microprofile/Uy-TwXuCEaM/2MfX5FzzCwAJ

The net of the conversation seemed to be that we don't want to tie people to NoSQL databases, and there are a lot of stable and widely used RDBs out there.  I fully agree with this conclusion that we don't want to tie MP to NoSQL or RDBs.

This proposal aims to define a lightweight spec that would build on top of _any_ existing or future data access technology (e.g JPA, JNoSQL, JDBC, or vendor-specific APIs such as MongoDB), and it would be up to the vendor implementation which data access technologies they want to build on top of (similar to how JPA impls can choose what RDBs they work with).

[Michael Musgrove]

The MicroProfile LRA project provides a description of the protocol: https://github.com/eclipse/microprofile-lra/blob/master/spec/src/main/asciidoc/microprofile-lra-spec.adoc.

The project is still in its early phase so we are looking for input/members to ensure that it meets the needs of the community and that the protocol description is unambiguous (I posted a message earlier that we are having a hangout next Monday). I have a Thorntail (this is the RedHat MP) implementation of it in my own repo which leverages the implementation of the spec that resides in the Narayana repo (https://github.com/jbosstm/narayana/tree/master/rts/lra). My plan is to release an (unstable) Thorntail fraction in the coming months as we get closer to finalising version 1 of the LRA specification. We also have a quickstart (https://github.com/jbosstm/quickstart/tree/master/rts/lra) for the narayana implementation.

And for background on where the project came from, there was a post on this group with the title "The need for transactions in the MicroProfile" (https://groups.google.com/forum/#!searchin/microprofile/long$20running$20actions$20musgrove%7Csort:date/microprofile/CJirjFkM9Do/8QdYMo-vAQAJ) that included some links to the protocol (specifically JDI which describes the protocol in terms of REST endpoints).

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/CABY0rKOvMpNaQMKYR0jJDDg8f6b28VFYKE92YXECXxZLXUdoCA%40mail.gmail.com.

[Michael Musgrove]

Nathan,

I see that you have not discussed in any detail which type of behaviours you would expect from microservices around atomicity, consistency or isolation? In one of my earlier posts I did speculatively suggest a set of annotations that would support these properties but it introduced too much coupling between services so I abandoned the proposal. I would be interested to hear your views on what would be useful in this regard.

[Gunnar Morling]

Hi James,

This thread is going into many directions (lots of great thoughts in here!), but I wanted to get back to this one point of yours.

> So capturing data updates requires reverse engineering the updates to work out what the event was

Indeed this can be an issue and we're looking into ways of helping developers here. I don't think it's always needed to store the full event. From the use cases and feedback I've seen, it'd often already help to just have a bit of contextual information available and linkable to change events. E.g. a business user applying a certain change, a use case identifier or something like that.

One way would be to just store that information within the actual tables. In that case, no help of any framework would be needed, it'd just be the user's task to provide that info. Change events would be implicitly self-contained that way. On the downside, there's an impact on the data model, which may not be desirable.

An alternative would be to just have a simple context table which stores given contextual information per transactions. Then change events (which have the TX id) can reference the corresponding contextual entries and be enriched with them. A data access API would have to be designed in a way so that a user can provide the contextual information when saving entities (similar to what you propose with passing the full events).

This might be an interesting middle-ground and I'm eager to explore that, e.g. by adding the required capabilities to Hibernate ORM so we could get hold of contextual information when mining change events from the database's log.

--Gunnar

2018-06-07 5:01 GMT+02:00 James Roper <ja...@lightbend.com>:

On Wed, 6 Jun 2018 at 23:09, 'Gunnar Morling' via Eclipse MicroProfile <microp...@googlegroups.com> wrote:

Nathan, James, all,

Thanks for starting this thread, it's a very important topic for MicroProfile. There are many interesting questions/suggestions in here and I hope to be able to comment on a few more, but one thing specifically caught my attention:

> 2) Complementary event log. In this, you have a traditional CRUD schema,

> but with every update you do to your schema, you also persist an event to 

> the event log in the same transaction. This event log can then be published 

> to a message broker without needing transactions, and updates can be 

> propagated with at least once messaging guarantees to other services.

This sounds very similar to what we do with CDC (change data capturing) in general and the Debezium (http://debezium.io/) project in particular. Only that the event log is in fact the one used maintained by the DB itself and that then Debezium connects to that log and creates change data streams based on table row changes. So the event steam comes "for free" for upstream applications, they don't have to do anything for that other than writing to the DB tables. Also in that approach there's a single source of truth (the DB's log) which is exposed in a (largely) uniform fashion by Debezium.

I think it's a great pattern for pushing data changes between microservices and I'd be very glad to help and propagate it in the context of MicroProfile.

Automatic capturing of updates and publishing them as messages has a lot of strengths, in particular, it's not prone to developer error in that developers can't forget to manually publish an event. But it also has a major drawback - when you capture the insert/update/delete done on the database, you lose the intention of the event. For example, let's say that the status of an order goes to "review". What was the event that triggered that? Did the payment processor reject the payment, and so now someone needs to review it? Did the inventory service say that the order can't be fulfilled, and so now someone needs to review it? Depending on what that intention was, different services that receive the event are going to need to do different processing, for example, an email notification service is going to send emails to different people with different wording based on what the event was, not based on what the data update was. The event is more than just the update, and knowing what the event is is important because other services might need to do different processing based on which event was received.

So capturing data updates requires reverse engineering the updates to work out what the event was, and this is both error prone, and in some circumstances requires a lot of code to get right. In the end, you're going to be converting these updates to some event and passing them to an API to be published, it's much easier to do this when you are handling the command in the first place and know exactly what the intention was and so you can publish the right event. That's why I think an API where you manually publish the event is much simpler for developers to use, and a well written API can offer protections to ensure that they don't forget to emit an event when doing updates, by, for example, requiring to emit an event before an update is committed.

--Gunnar

--

You received this message because you are subscribed to the Google Groups "Eclipse MicroProfile" group.
To unsubscribe from this group and stop receiving emails from it, send an email to microprofile...@googlegroups.com.
To post to this group, send email to microp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/7aa8ced6-d969-4bfb-8853-12db86a08b7b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

James Roper
Senior Developer, Office of the CTO

Lightbend – Build reactive apps!
Twitter: @jroper

--
You received this message because you are subscribed to the Google Groups "Eclipse MicroProfile" group.

To unsubscribe from this group and stop receiving emails from it, send an email to microprofile+unsubscribe@googlegroups.com.

To post to this group, send email to microp...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/29b5c66f-3c78-4727-8072-d12f649120d3%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

James Roper
Senior Developer, Office of the CTO

Lightbend – Build reactive apps!
Twitter: @jroper

--

You received this message because you are subscribed to a topic in the Google Groups "Eclipse MicroProfile" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/microprofile/1sXozNjXLsM/unsubscribe.

To unsubscribe from this group and all its topics, send an email to microprofile+unsubscribe@googlegroups.com.

To post to this group, send email to microp...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/CABY0rKO6M3tvQ5F2P2mmt2Rk3fFqOPXPB%3DdW5TrByytvcOK6gA%40mail.gmail.com.

[Michael Musgrove]

James,

The microprofile-lra spec is really just a collection of java annoations for controlling the lifecycle of LRAs and participants with some guarantees. I do not imagine there would be any barriers to using you CQRS event sourcing approach to implement the specification, If it isn't then the expedient approach would be to update the spec,

On Fri, Jun 15, 2018 at 5:18 PM James Roper <ja...@lightbend.com> wrote:

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/CABY0rKOvMpNaQMKYR0jJDDg8f6b28VFYKE92YXECXxZLXUdoCA%40mail.gmail.com.

[Tom Jenkinson]

Perhaps it would be an interesting topic to add to the agenda for the Microprofile LRA hangout on Monday?

[Nathan Rauh]

Michael,

While I left out this sort of detail from the original idea (I knew the transactional aspects would be controversial), it is essential for a data access spec to be capable of addressing the variety of patterns that will be needed for different scenarios.  In the most basic of patterns where data access is self-contained within a service and never coordinated with any other services, there just needs to be a way to demarcate transaction boundaries within the service itself, and the traditional UserTransaction begin/commit/rollback ought to be sufficient for that.  Although many will hate to hear me say it, I think there's even a case for distributed two-phase transactions across services for the specific patterns where the need to coordinate across services is very infrequent.  As for patterns that address the coordination of independent services, I can see a data access spec seamlessly tying in to the Long Running Actions spec.

Data access template update/insert/delete operations provide a convenient point to automatically detect the presence of the current LRA and join it, computing the detail of the changes incurred by the operation and persisting this information to the same database within the same transaction, implicitly generating a compensation action that is capable of using this information to undo/revert the operation.  This spares the user from the biggest burden of using long running actions - providing compensation actions - allowing them to focus solely on demarcating start/completion of the LRA and the services that participate.  The user would be able to utilize CRUD operations within their service just as they would when LRA was not involved at all.  The data access spec and LRA spec would remain independent, but seamlessly interoperable.

The above is one reason why to propose data access as a separate Microprofile spec rather than as updates to an existing data access spec like JPA.  JPA being a Jakarta EE technology, cannot define how it should integrate with Microprofile technologies.

Nathan Rauh
____________________________________________
Software Engineer, WebSphere Application Server
IBM Rochester Bldg 050-2 H215-1
3605 Highway 52N
Rochester, MN 55901-7802

From:        Michael Musgrove <michael....@gmail.com>
To:        Eclipse MicroProfile <microp...@googlegroups.com>
Date:        06/15/2018 11:39 AM
Subject:        [microprofile] Re: Data access spec for MicroProfile
Sent by:        microp...@googlegroups.com

---

--
You received this message because you are subscribed to a topic in the Google Groups "Eclipse MicroProfile" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/microprofile/1sXozNjXLsM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to microprofile...@googlegroups.com.

To post to this group, send email to

microp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/2b70dae4-2a86-48e9-8f61-c567da445665%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Emily Jiang]

Nathan,

I thought this a bit more. I think in order to make further progress, some fundamental questions need to be nailed first e.g. how best to handle the data persistence in microservice world. These should be defined before even defining data access APIs. If this basic persistence model is not agreed, defining a data access API might not be very useful.

We need to consider some basic use case: e.g.

service a -> service b,
how data are persisted in two databases (one database per microservice). How does the rollback model work? I assume most of time data persistence should be in a transaction. Hence, some transaction context should be defined, which might impact data persistence. I think this use case should also link with reactive as persisting data might be better defined as non-blocking etc.

Therefore, I suggest a conversation with lra and reactive might be a good start.

Thanks

Emily

.
To unsubscribe from this group and all its topics, send an email to microprofile+unsubscribe@googlegroups.com.

To post to this group, send email to

microp...@googlegroups.com.

To view this discussion on the web visit

https://groups.google.com/d/msgid/microprofile/2b70dae4-2a86-48e9-8f61-c567da445665%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Eclipse MicroProfile" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/microprofile/1sXozNjXLsM/unsubscribe.

To unsubscribe from this group and all its topics, send an email to microprofile+unsubscribe@googlegroups.com.

To post to this group, send email to microp...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/OF59219F6E.E12919EB-ON002582B3.00517D46-872582B3.005DB3F9%40notes.na.collabserv.com.

For more options, visit https://groups.google.com/d/optout.

--

Thanks
Emily
=================
Emily Jiang
eji...@apache.org

[Tom Jenkinson]

On Thursday, June 21, 2018 at 5:03:36 PM UTC+1, Nathan Rauh wrote:

Michael,

While I left out this sort of detail from the original idea (I knew the transactional aspects would be controversial),

:) for saga-related extended transaction models I am not much criticism about their usage. On the other hand, I am aware that Camel do already have a feature that uses them: https://github.com/apache/camel/blob/master/camel-core/src/main/docs/eips/saga-eip.adoc

 

it is essential for a data access spec to be capable of addressing the variety of patterns that will be needed for different scenarios.  In the most basic of patterns where data access is self-contained within a service and never coordinated with any other services, there just needs to be a way to demarcate transaction boundaries within the service itself, and the traditional UserTransaction begin/commit/rollback ought to be sufficient for that.  Although many will hate to hear me say it, I think there's even a case for distributed two-phase transactions across services for the specific patterns where the need to coordinate across services is very infrequent.  As for patterns that address the coordination of independent services, I can see a data access spec seamlessly tying in to the Long Running Actions spec.

Data access template update/insert/delete operations provide a convenient point to automatically detect the presence of the current LRA and join it, computing the detail of the changes incurred by the operation and persisting this information to the same database within the same transaction, implicitly generating a compensation action that is capable of using this information to undo/revert the operation.  This spares the user from the biggest burden of using long running actions - providing compensation actions - allowing them to focus solely on demarcating start/completion of the LRA and the services that participate.  The user would be able to utilize CRUD operations within their service just as they would when LRA was not involved at all.  The data access spec and LRA spec would remain independent, but seamlessly interoperable.

This elements sounds interesting, are you able to elaborate on how the "undo" operation could be reliably generated?

For some types of application inverting the original operation might not be possible or the preferred outcome which, as you point out, does place a requirement on the application programmer to consider the possibility ahead of time but providing an option to generate an undo operation for the user would be a useful option.

If you would like to chat about the way transactions could interact with your proposal I know there is a Microprofile LRA hangout which Mike hosts (details are over here: https://groups.google.com/forum/#!topic/microprofile/5tIZHUPyLXM)

.
To unsubscribe from this group and all its topics, send an email to microprofile+unsubscribe@googlegroups.com.

To post to this group, send email to

microp...@googlegroups.com.

To view this discussion on the web visit

https://groups.google.com/d/msgid/microprofile/2b70dae4-2a86-48e9-8f61-c567da445665%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Emmanuel Bernard]

Answering to this post and the following ones.

I think there are several somewhat parallel goal clusters that might be achieved by one or several specs / work.

Addressing the multi microservices persistence (or rather state transfer) problem. This is where LRA, EventSourcing, Change Data Capture make the most sense.

Addressing the simplification of (JPA) data access. This is where template, query description and easier bootstrap show up.

I understand the JPA in limbo concern but I don't fully share it. Hopefully Jakarta EE will bring this in motion. And I think 2/3 of this proposal can be built on top of JPA without needing it. We could explore with specific JPA providers ahead of time.

Addressing abstraction of any/all data persistence problems.

My personal analysis would point that most (say 95%+) of Spring Data usages are Spring Data JPA or Spring Data JDBC and I suppose the big majority is for JPA.

Spring Data does not pretend to offer a NoSQL abstraction, *Very* quickly you have to rely on backend specific (Spring Data) APIs if not straight to the raw native API of the backend. It's not bad, but let's not pretend such/just the abstraction addresses the problem.

While we should define the whole picture, I would not bring these three main areas under the same spec / basket.

Emmanuel

[Nathan Rauh]

Tom,
Good question. Here are some more detailed thoughts on auto-generating LRA compensator actions.

First, it needs to be clarified it's not possible to automatically compute a compensation action for every sort of update.  I wasn't clear on that in the previous message.  But generalized handling can apply to some common core scenarios.  This includes updates that directly perform increment/decrement, those that indirectly increment/decrement (for example, x = x * 1.03), and to some degree assignment of values (for example, s = "STARTED").  The former two will generally be revertible. The latter will be revertible only if the value has not subsequently been changed by another LRA.  In cases where generalizations are not suitable, the user would fall back on the normal LRA approach of providing their own compensation action.  Anyone who is reading this can chime in with additional areas where generic handling can apply.  Hopefully others will be thought of.

Here is an example of how it could work for the simple case,

db.update(1, quantity.decrement(10));     // SET quantity = quantity - 10 WHERE id = 1
  write entry with LRU identifier and data,
    { "id": 1, "diff": { "quantity": { "op": "-", "val": 10 } } }, ...
  perform the update as usual
  commit if successful

  compensator finds & removes record, then
    UPDATE ... SET quantity = quantity + 10 WHERE id = 1
    commit if successful


An example with multiple rows updated and values computed based on another column,

db.update(id.greaterThan(0).and(id.lessThan(3)), a.decrement(b));  // SET a = a - b WHERE id > 0 AND id < 3
  first obtain values of b by performing query with update intent
    SELECT id, a, b FROM mytable WHERE id = 1 OR id = 2 FOR UPDATE
  write entry with LRU identifier and data, (where V1 is computed as b for id of 1)
    { "id": 1, "diff": { "a": { "op": "-", "val": <V1> } } }
  perform update for id=1
  write entry with LRU identifier and data, (where V2 is computed as b for id of 2)
    { "id": 2, "diff": { "a": { "op": "-", "val": <V2> } } }
  perform update for id=2
  commit if all successful

  compensator finds & removes record, then
    UPDATE ... SET a = a + <V1> WHERE id = 1
    UPDATE ... SET a = a + <V2> WHERE id = 2
    commits if all successful


An example with multiply & assign treated as increment/decrement,

db.update(1, a.notEq(0), a.increment(a.mult(0.05)));  // SET a = a + a * 0.05 WHERE id = 1 AND a <> 0
  first obtain value of a by performing query with update intent
    SELECT id, a FROM mytable WHERE id = 1 AND a <> 0
  write entry with LRU identifier and data, (where V1 = a * 0.05)
    { "id": 1, "diff": { "a": { "op": "+", "val": <V1> } } }
  perform update
  commit if successful

  compensator finds & removes record, then
    UPDATE ... SET a = a - <V1> WHERE id = 1
    commits if successful


An example with assignment rather than increment/decrement,

db.update(1, b.greaterThan(c), a.assign(b.minus(c)), d.assign(null));  // SET a = b - c, d = null WHERE id = 1 AND b > c
  first obtain values of a, b, c, d by performing query with update intent
    SELECT id, a, b, c, d FROM mytable WHERE id = 1 AND b > c
  write entry with LRU identifier and data, (where V1 = b - c, V2 is the old value of 'a', and V3 is the old value of 'd')
    { "id": 1, "diffs": { "a": { "op": "=", "val": <V1>, "prev": <V2> }, "d": { "op": "=", "val": null, "prev": <V3> } } }
  perform update
  commit if successful

  compensator finds & removes record, then
    UPDATE ... SET a = <V2>, d = <V3> WHERE id = 1 AND a = <V1> AND d = NULL
    commits if successful


As part of the transaction that performs the revert, the compensation data is also removed. This ensures that the revert will be committed at most once.

There might be cases where, although the compensation action is properly computed, it may be undesirable to apply it (for example, causing a balance to go negative because the addition that we are trying to compensate/revert was all or partly used by a different LRA).  I could see adding a way to denote constraints to address these.  In the case of reverting and assignment, the current value might no longer match because another LRA has since updated it.
In both of these cases (and likely others), the compensate/revert action will fail.  I'm unsure how LRA is able to handle the FailedToCompensate scenario, but these sort of paths belong under it.

.
To unsubscribe from this group and all its topics, send an email to microprofile...@googlegroups.com.

To post to this group, send email to

microp...@googlegroups.com.

To view this discussion on the web visit

https://groups.google.com/d/msgid/microprofile/2b70dae4-2a86-48e9-8f61-c567da445665%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to a topic in the Google Groups "Eclipse MicroProfile" group.
To unsubscribe from this topic, visit

https://groups.google.com/d/topic/microprofile/1sXozNjXLsM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to microprofile...@googlegroups.com.

To post to this group, send email to

microp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/660528f0-898c-4ecd-a685-890a2259e3e2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Nathan Rauh]

Addressing some questions from Emily,

> How does the rollback model work?

Rollback of a data access transaction must result in a request to cancel the LRA so that compensation can be performed for any prior operations.  The data access layer, which will be aware of the LRA, will make this request.

> Hence, some transaction context should be defined, which might impact data persistence.

There are two forms of transaction context that we are concerned with: context for the actual transaction in the database, and context for the LRA which coordinates the transaction with transactions being performed by other services.
For the former, we need a way to group transactional operations that do not cross service boundaries.  There ought to be some discussion of whether the existing JTA UserTransaction is the right way to do that, or if data access spec should be defining its own way to either group multiple of its own operations into a transaction or demarcate a begin/end.
For the latter (LRA context), it falls under the scope of the LRA spec, not data access, to define how that context is managed.  I could see it being useful to have integration between LRA and concurrency utilities specs that allows for propagating LRA context to completion stages and executor tasks.

> link with reactive as persisting data might be better defined as non-blocking etc.

I can think of a couple of reactive patterns that we should consider.

One possibility is incorporating CompletionStage return values to simulate non-blocking usage.
For example,
    CompletionStage<Boolean> stage1 = db.updateAsync(key1, ...);
    CompletionStage<Boolean> stage2 = db.updateAsync(key2, ...)
        .thenCombine(stage1, (u2, u1) -> u2 || u1);
However, even without such a pattern, the following would be possible,
    CompletionStage<Boolean> stage1 = completableFuture.supplyAsync(() -> db.update(key1, ...));
    CompletionStage<Boolean> stage2 = completableFuture.supplyAsync(() -> db.update(key2, ...))
        .thenCombine(stage2, (u2, u1) -> u2 || u1);
These are similar enough that I didn't see a compelling reason to include this sort of convenience method.  There is another reason to consider it, though, which I'll get to later.

The other pattern to consider is a method variant that allows a subscriber to be supplied. For example,
db.find(query, subscriber)
But is this even the right level to have a subscriber?  Would it be better done by the service itself while processing the stream of results, for example enabling the service to filter out or transform some?

Is anyone aware of other reactive patterns that we should be considering?

Getting back to the discussion of CompletionStage, I mentioned there was another benefit that having template operations return a CompletionStage would give us, although I don't believe it is the right long term design.  This is that these async operations really ought to be running on container managed threads, for several reasons:
* Access to java: namespace for the application, which might end up being necessary if accessing the underlying resource via a resource reference
* Propagation of the LRA context of the submitter so that operations performed in dependent stages can run under the same LRA as the thread which requests them.
* Propagation of security credentials of the submitter so that operations performed in dependent stages can run under the same security context as the thread which requests them.
* Usage of a container managed thread pool
The proper design for this is not to have Data Access templates provide such a CompletionStage.  Instead, it would best be done as an update to the Jakarta EE concurrency utilities spec that provides a CompletionStage/CompletableFuture with these abilities given a managed executor service.  This might be something that a Microprofile spec (not Data Access) could provide in the mean time.  Or it could be considered a future direction that we want to eventually get to once Jakarta EE catches up.

.
To unsubscribe from this group and all its topics, send an email to microprofile...@googlegroups.com.

To post to this group, send email to

microp...@googlegroups.com.

To view this discussion on the web visit

https://groups.google.com/d/msgid/microprofile/2b70dae4-2a86-48e9-8f61-c567da445665%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Eclipse MicroProfile" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/microprofile/1sXozNjXLsM/unsubscribe

.
To unsubscribe from this group and all its topics, send an email to microprofile...@googlegroups.com.

To post to this group, send email to

microp...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/OF59219F6E.E12919EB-ON002582B3.00517D46-872582B3.005DB3F9%40notes.na.collabserv.com.
For more options, visit https://groups.google.com/d/optout.

--
Thanks
Emily
=================
Emily Jiang
eji...@apache.org

--

You received this message because you are subscribed to a topic in the Google Groups "Eclipse MicroProfile" group.
To unsubscribe from this topic, visit

https://groups.google.com/d/topic/microprofile/1sXozNjXLsM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to microprofile...@googlegroups.com.

To post to this group, send email to

microp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/CAECq3A-%3D9ekNvULOy_NQVNKpNt7UjTJyB%2BG_YqOiYkDNexTtqA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Tom Jenkinson]

On 27 June 2018 at 15:30, Nathan Rauh <natha...@us.ibm.com> wrote:

Addressing some questions from Emily,

> How does the rollback model work?
Rollback of a data access transaction must result in a request to cancel the LRA so that compensation can be performed for any prior operations.  The data access layer, which will be aware of the LRA, will make this request.

> Hence, some transaction context should be defined, which might impact data persistence.
There are two forms of transaction context that we are concerned with: context for the actual transaction in the database, and context for the LRA which coordinates the transaction with transactions being performed by other services.
For the former, we need a way to group transactional operations that do not cross service boundaries.  There ought to be some discussion of whether the existing JTA UserTransaction is the right way to do that, or if data access spec should be defining its own way to either group multiple of its own operations into a transaction or demarcate a begin/end.

I would hope that JTA could be used within individual microservices. JTA has well defined semantics that MicroProfile can benefit from. It does assume though that individual resources (even the NoSQL ones for example) can provide an XAResource, is that a reasonable expectation for resources we might expect to be used in a MicroProfile environment?

 

.
To unsubscribe from this group and all its topics, send an email to microprofile+unsubscribe@googlegroups.com.

To post to this group, send email to

microp...@googlegroups.com.

To view this discussion on the web visit

https://groups.google.com/d/msgid/microprofile/2b70dae4-2a86-48e9-8f61-c567da445665%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Eclipse MicroProfile" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/microprofile/1sXozNjXLsM/unsubscribe

.
To unsubscribe from this group and all its topics, send an email to microprofile+unsubscribe@googlegroups.com.

To post to this group, send email to

microp...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/OF59219F6E.E12919EB-ON002582B3.00517D46-872582B3.005DB3F9%40notes.na.collabserv.com.
For more options, visit https://groups.google.com/d/optout.

--
Thanks
Emily
=================
Emily Jiang
eji...@apache.org

--
You received this message because you are subscribed to a topic in the Google Groups "Eclipse MicroProfile" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/microprofile/1sXozNjXLsM/unsubscribe

.
To unsubscribe from this group and all its topics, send an email to microprofile+unsubscribe@googlegroups.com.

To post to this group, send email to

microp...@googlegroups.com

.

To view this discussion on the web visit

https://groups.google.com/d/msgid/microprofile/CAECq3A-%3D9ekNvULOy_NQVNKpNt7UjTJyB%2BG_YqOiYkDNexTtqA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "Eclipse MicroProfile" group.

To unsubscribe from this group and stop receiving emails from it, send an email to microprofile+unsubscribe@googlegroups.com.

To post to this group, send email to microp...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/OF1DE71865.BEC5E673-ON002582B9.0048DBEE-862582B9.004FB4CD%40notes.na.collabserv.com.

[Nathan Rauh]

Tom,

That's a very good point to consider.  Most NoSQL databases that I'm aware of do not provide XAResource, and so usage of JTA API to demarcate begin/commit/rollback for those would involve the data access layer enlisting a resource on their behalf that is only one-phase capable, relying on something like last participant support should other resources be enlisted in the transaction as well.  Beside this, it should be noted that the ability to group operations into a transactions is subject to the database's ability to support it (many NoSQL databases don't).  The data access layer, being aware of the presence of a transaction (however it is denoted) should be required to reject an operation that does not support enlisting with a transaction that is present.

By deciding to use JTA as the mechanism to group multiple operations into a transaction, we would need to be okay with accomodations along the lines of the above for databases that don't provide XAResource.

Nathan Rauh
____________________________________________
Software Engineer, WebSphere Application Server
IBM Rochester Bldg 050-2 H215-1
3605 Highway 52N
Rochester, MN 55901-7802

.
To unsubscribe from this group and all its topics, send an email to microprofile...@googlegroups.com.

To post to this group, send email to

microp...@googlegroups.com.

To view this discussion on the web visit

https://groups.google.com/d/msgid/microprofile/2b70dae4-2a86-48e9-8f61-c567da445665%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Eclipse MicroProfile" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/microprofile/1sXozNjXLsM/unsubscribe

.
To unsubscribe from this group and all its topics, send an email to microprofile...@googlegroups.com.

To post to this group, send email to

microp...@googlegroups.com.

To view this discussion on the web visit

https://groups.google.com/d/msgid/microprofile/OF59219F6E.E12919EB-ON002582B3.00517D46-872582B3.005DB3F9%40notes.na.collabserv.com.
For more options, visit https://groups.google.com/d/optout.

--
Thanks
Emily
=================
Emily Jiang
eji...@apache.org

--
You received this message because you are subscribed to a topic in the Google Groups "Eclipse MicroProfile" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/microprofile/1sXozNjXLsM/unsubscribe

.
To unsubscribe from this group and all its topics, send an email to microprofile...@googlegroups.com.

To post to this group, send email to

microp...@googlegroups.com

.
To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/CAECq3A-%3D9ekNvULOy_NQVNKpNt7UjTJyB%2BG_YqOiYkDNexTtqA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Eclipse MicroProfile" group.

To unsubscribe from this group and stop receiving emails from it, send an email to microprofile...@googlegroups.com.

To post to this group, send email to

microp...@googlegroups.com.

To view this discussion on the web visit

https://groups.google.com/d/msgid/microprofile/OF1DE71865.BEC5E673-ON002582B9.0048DBEE-862582B9.004FB4CD%40notes.na.collabserv.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to a topic in the Google Groups "Eclipse MicroProfile" group.
To unsubscribe from this topic, visit

https://groups.google.com/d/topic/microprofile/1sXozNjXLsM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to microprofile...@googlegroups.com.

To post to this group, send email to

microp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/CALaezvXYb7RK%2Ba3CxTsaKV9qRY9X79hLM-P_M6m9yMgZ9__o1A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Tom Jenkinson]

Out of the box, JTA is only designed to interact reliably with resource managers that are able to define XA resources and using last resource commit is not covered by the specification itself. Application servers typically provide that functionality in a proprietary manner and with caveats as to what can be relied upon from it. Coincidentally there is a discussion at the moment ongoing in the JTA spec that relates somewhat to this (it is particularly about ordering of resources in a transaction, but there is reference to 1PC): https://github.com/javaee/jta-spec/issues/4#issuecomment-400353377

Perhaps the data access specification could indicate support for various transaction models:

1. Standard ACID (when the resource has an XAResource or at least something that exposes a prepare/commit in the driver)

2. Compensation based - either using a similar generator pattern as you proposed for the service layer or to be provided by the developer - would need to rely on idempotent updates.

If we were to relax isolation, perhaps the second approach might be achieved as follows:

TransactionManager tm = ;//

tm.enlistResource(xaResource = new DataAccessSpecXAResource(noSQLdatasource))

DataAccesSpecResource.update(//cache them locally rather than flush to server)

tm.commit() {

  xaResource.prepare(// DataAccesSpecResource makes durable identifying information about the changes to make to allow for post-crash commit calls)

  xaResource.commit(// DataAccesSpecResource pushes changes to remote database, if the resource is not allowing muliple updates in one TX then the updates need to be idempotent)

  xaResource.rollback(// DataAccessSpecResource runs "compensation" routine, this logic needs to be able to deal with cases where the original update was never flushed)

}

.
To unsubscribe from this group and all its topics, send an email to microprofile+unsubscribe@googlegroups.com.

To post to this group, send email to

microp...@googlegroups.com.

To view this discussion on the web visit

https://groups.google.com/d/msgid/microprofile/2b70dae4-2a86-48e9-8f61-c567da445665%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Eclipse MicroProfile" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/microprofile/1sXozNjXLsM/unsubscribe

.
To unsubscribe from this group and all its topics, send an email to microprofile+unsubscribe@googlegroups.com.

To post to this group, send email to

microp...@googlegroups.com.

To view this discussion on the web visit

https://groups.google.com/d/msgid/microprofile/OF59219F6E.E12919EB-ON002582B3.00517D46-872582B3.005DB3F9%40notes.na.collabserv.com.
For more options, visit https://groups.google.com/d/optout.

--
Thanks
Emily
=================
Emily Jiang
eji...@apache.org

--
You received this message because you are subscribed to a topic in the Google Groups "Eclipse MicroProfile" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/microprofile/1sXozNjXLsM/unsubscribe

.
To unsubscribe from this group and all its topics, send an email to microprofile+unsubscribe@googlegroups.com.

To post to this group, send email to

microp...@googlegroups.com

.
To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/CAECq3A-%3D9ekNvULOy_NQVNKpNt7UjTJyB%2BG_YqOiYkDNexTtqA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Eclipse MicroProfile" group.

To unsubscribe from this group and stop receiving emails from it, send an email to microprofile+unsubscribe@googlegroups.com.

To post to this group, send email to

microp...@googlegroups.com.

To view this discussion on the web visit

https://groups.google.com/d/msgid/microprofile/OF1DE71865.BEC5E673-ON002582B9.0048DBEE-862582B9.004FB4CD%40notes.na.collabserv.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Eclipse MicroProfile" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/microprofile/1sXozNjXLsM/unsubscribe

.
To unsubscribe from this group and all its topics, send an email to microprofile+unsubscribe@googlegroups.com.

To post to this group, send email to

microp...@googlegroups.com

.

To view this discussion on the web visit

https://groups.google.com/d/msgid/microprofile/CALaezvXYb7RK%2Ba3CxTsaKV9qRY9X79hLM-P_M6m9yMgZ9__o1A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Eclipse MicroProfile" group.

To unsubscribe from this group and stop receiving emails from it, send an email to microprofile+unsubscribe@googlegroups.com.

To post to this group, send email to microp...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/OF2A1BD8E5.3935FC40-ON002582B9.00562662-862582B9.0057AE3B%40notes.na.collabserv.com.

[Nathan Rauh]

Tom,

+1 to the JTA spec update and your other comments.  It would be great to see ordering options standardized to accommodate last participant without vendor-specific code.

To elaborate on the two transaction models, we should add a variant of the first model (labeled as 1a. below) that doesn't require the driver to be capable of a prepare at all,

> 1. Standard ACID (when the resource has an XAResource or at least something that exposes a prepare/commit in the driver)

1a. Standard ACID with one phase optimization - In this case, UserTransaction.begin/commit/rollback are only being used as a way to demarcate transaction boundaries for a single participant, such that transactional operations on that single resource are grouped together.  For example,  tran.begin(); template.update(key1, ...); template.update(key2, ...); tran.commit();  As long as the driver provides a way to begin and commit a transaction (no prepare necessary), the data access layer should be able to enlist an XAResource of its own that rejects any attempt at xa.prepare and only allows xa.commit(onePhase=true), which the transaction manager uses upon seeing that only a single resource was enlisted.  This pattern is covered in the XA spec under 2.3.2 Protocol Optimizations | One-phase Commit.

1b. Standard ACID with two phase commit - corresponds to what you described where prepare is needed.

For completeness, there is also the most basic case,
1c. Standard ACID where data access operation is performed without any explicit transaction boundary or LRA present, which is expected to commit upon completion of the update/insert/remove operation in the database.

.
To unsubscribe from this group and all its topics, send an email to microprofile...@googlegroups.com.

To post to this group, send email to

microp...@googlegroups.com.

To view this discussion on the web visit

https://groups.google.com/d/msgid/microprofile/2b70dae4-2a86-48e9-8f61-c567da445665%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Eclipse MicroProfile" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/microprofile/1sXozNjXLsM/unsubscribe

.
To unsubscribe from this group and all its topics, send an email to microprofile...@googlegroups.com.

To post to this group, send email to

microp...@googlegroups.com.

To view this discussion on the web visit

https://groups.google.com/d/msgid/microprofile/OF59219F6E.E12919EB-ON002582B3.00517D46-872582B3.005DB3F9%40notes.na.collabserv.com.
For more options, visit https://groups.google.com/d/optout.

--
Thanks
Emily
=================
Emily Jiang
eji...@apache.org

--
You received this message because you are subscribed to a topic in the Google Groups "Eclipse MicroProfile" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/microprofile/1sXozNjXLsM/unsubscribe

.
To unsubscribe from this group and all its topics, send an email to microprofile...@googlegroups.com.

To post to this group, send email to

microp...@googlegroups.com

.
To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/CAECq3A-%3D9ekNvULOy_NQVNKpNt7UjTJyB%2BG_YqOiYkDNexTtqA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Eclipse MicroProfile" group.

To unsubscribe from this group and stop receiving emails from it, send an email to microprofile...@googlegroups.com.

To post to this group, send email to

microp...@googlegroups.com.

To view this discussion on the web visit

https://groups.google.com/d/msgid/microprofile/OF1DE71865.BEC5E673-ON002582B9.0048DBEE-862582B9.004FB4CD%40notes.na.collabserv.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Eclipse MicroProfile" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/microprofile/1sXozNjXLsM/unsubscribe

.
To unsubscribe from this group and all its topics, send an email to microprofile...@googlegroups.com.

To post to this group, send email to

microp...@googlegroups.com

.
To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/CALaezvXYb7RK%2Ba3CxTsaKV9qRY9X79hLM-P_M6m9yMgZ9__o1A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Eclipse MicroProfile" group.

To unsubscribe from this group and stop receiving emails from it, send an email to microprofile...@googlegroups.com.

To post to this group, send email to

microp...@googlegroups.com.

To view this discussion on the web visit

https://groups.google.com/d/msgid/microprofile/OF2A1BD8E5.3935FC40-ON002582B9.00562662-862582B9.0057AE3B%40notes.na.collabserv.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to a topic in the Google Groups "Eclipse MicroProfile" group.
To unsubscribe from this topic, visit

https://groups.google.com/d/topic/microprofile/1sXozNjXLsM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to microprofile...@googlegroups.com.

To post to this group, send email to

microp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/CALaezvVt6NY7jXYp8JUOH9N6Y%2Bm_8YrF7LVEkSiv9OkSnboZ4g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.