Reactive Relational DB Access in MicroProfile

[Andy Guibert]

Hi all,

Recently there have been efforts within MP to define a standard set of reactive APIs, namely MP Reactive Streams. When considering relational DB access, there are a number of solid options today, such as JPA, Hibernate, MyBatis, or jOOQ. However, all of these options are based on JDBC, which is a blocking API.

When we look at _reactive_ relational DB access, there are two main efforts brewing:

• R2DBC (Reactive Relational DB Connectivity, open source, driven by Pivotal)
• ADBA (Async DB Access, internally designed, driven by Oracle)

Neither of these efforts are officially supported yet, but R2DBC looks like it's close. Additionally, it has the advantage over ADBA that it's a library, as opposed to ADBA which is proposed for inclusion in the JDK (and won't be included until JDK 13 at the absolute earliest).

I think it would be interesting to experiment with combinations of R2DBC and MP Reactive Streams to see if we can come up with an API that allows applications to access relational DBs in a simple, extensible, and reactive way.

Anyone else interested in collaborating on this, or have further thoughts/questions?

Thanks,

Andy Guibert

[m.reza.rahman]

It is an overdue idea. Glad we are looking at it. If you need contacts to sync with the Oracle folks, let me know.

Sent via the Samsung Galaxy S7, an AT&T 4G LTE smartphone

--
You received this message because you are subscribed to the Google Groups "Eclipse MicroProfile" group.
To unsubscribe from this group and stop receiving emails from it, send an email to microprofile...@googlegroups.com.
To post to this group, send email to microp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/562c0232-45e4-4c50-95f4-b125889ec318%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ondro Mihályi]

This is a great idea and nicely fits into existing reactive efforts (operators and messaging).

From what I know about R2DBC and ADBA is that R2DBC is the only fully reactive API, with support for backpressure. ADBA is only asynchronous, so it's useful, but not complete as it doesn§t support flow control. That's basically the reason why Pivotal started R2DBC. Another drawback of ADBA is not just that it's a JDK effort and will require a certain version of JDK but that it's been talked about for years and there's still nothing available.

I'm all in to explore R2DBC and even cooperate with Pitoval folks on its development while working on an extension or a spec to integrate it well into MP.

Ondro

[Oliver Drotbohm]

Hi all,

great to hear you're looking into this. Please make sure to let us know in the dedicated R2DBC Google Group if there's anything we can do to help you pick this up easily!

Cheers,

Ollie

[Ondro Mihályi]

Hi Oliver,

Thanks for chiming in here and sharing info how to connect with the R2DBC group!

I think it makes a lot of sense to widely collaborate in this area which is framework neutral and very useful for the entire Java community.

Ondro

št 10. 1. 2019 o 9:48 Oliver Drotbohm <odro...@pivotal.io> napísal(a):

--
You received this message because you are subscribed to a topic in the Google Groups "Eclipse MicroProfile" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/microprofile/Em24P1UCcPE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to microprofile...@googlegroups.com.

To post to this group, send email to microp...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/514ce09e-565e-4f5d-ad02-84a2b5cada5e%40googlegroups.com.

[Gordon Hutchison]

This is a great initiative in my opinion.

Two things come up for me:

regarding R2DBC v ADBA

1) licensing??

E.g. the file at

http://hg.openjdk.java.net/jdk/sandbox/file/b49fd5dbaab6/src/java.sql/share/classes/java/sql2/RowProcessorOperation.java#l58

Has "GNU General Public License version 2 only"

and at http://mail.openjdk.java.net/pipermail/jdbc-spec-discuss/2018-October/000459.html

I see a discussion on problems on the GPL license of ADBA and pulling it into an

Apache 2.0 project.

But at this URL

https://www.oracle.com/technetwork/database/application-development/jdbc/sv-jug-adba-5022946.pdf

I see for ADBA

"Source released under the Apache license"

So go figure??

2) Secondly

I think we need to build this with org.reactivestreams elements for Publishers/Subscribers (or perhaps j.u.c.Flow in the future)

and move up the MicroProfile Reactive Stream *Builders classes for higher level classes in order to stay compatible

with other MicroProfile specs like messaging...rather than incorporating the use of Flux or Mono into MicroProfile.

I know the R2DBC SPI is Flux/Mono free and it is only the R2DBC Client API that is based on them

so we could base on the R2DBC SPI without using Flux/Mono. Also the 'isa' nature of Flux/Mono

does mean that they are, at the end of the day, also org.reactive-streams types so they can be

plumbed at that level - but there are advantages to following the idiom of the existing

PublisherBuilder, SubscriberBuilder types that are used in MicroProfile Reactive Streams and

MicroProfile Reactive Messaging as it allows for easier stream composition by the user

and easier (cross) vendor value add in running the stream.

Gordon Hutchison

[Oliver Drotbohm]

Hi Gordon,

> Am 11.01.2019 um 11:34 schrieb Gordon Hutchison <gordon.h...@gmail.com>:
>
> I know the R2DBC SPI is Flux/Mono free and it is only the R2DBC Client API that is based on them
> so we could base on the R2DBC SPI without using Flux/Mono. Also the 'isa' nature of Flux/Mono
> does mean that they are, at the end of the day, also org.reactive-streams types so they can be
> plumbed at that level - but there are advantages to following the idiom of the existing
> PublisherBuilder, SubscriberBuilder types that are used in MicroProfile Reactive Streams and
> MicroProfile Reactive Messaging as it allows for easier stream composition by the user
> and easier (cross) vendor value add in running the stream.

The idea behind the client API(s!) is that multiple exist that leverage the power of the ecosystem they run in. In the Spring world, R2DBC Client has been mostly superseded by Spring Data R2DBC. I think building something CDI based for Microprofile is the most promising way. IIRC there was some Spring Data inspired project in Deltaspike. Maybe it makes sense to get some inspiration there.

On the Reactive Streams VS. Reactor discussion you need to take note that Reactor is currently the only reactive library that exposes a Reactive Context (think of it as a ThreadLocal in the reactive world) that allows us to implement things like transactions. I.e. until other reactive libraries or APIs expose that kind of context, transactions will be hard (if not impossible) to implement and relational data access doesn't make too much sense without those.

Cheers,
Ollie

[Mark Paluch]

Hey there, 

 

only the R2DBC Client API that is based on (Flux|Mono) 

R2DBC Client is a JDBI-inspired client implementation putting a tiny layer of a more humane API on top of R2DBC SPI. 

R2DBC SPI does not aim to be used by users directly rather it provides a friendly API for client implementors.

Following the idea to stick with PublisherBuilder et al. it would make sense to get some inspiration from R2DBC SPI or R2DBC Client to expose R2DBC in a way that embeds smoothly into MicroProfile's Reactive API ecosystem.

Cheers, 

Mark

[Mark Paluch]

> Reactor is currently the only reactive library that exposes a Reactive Context (think of it as a ThreadLocal in the reactive world)

I don't think the context is required per se, it's handy in the way Spring sees transaction management. Transaction demarcation can be expressed through various means (e.g. callback closures) which makes it perfectly fine to not have an environmental scoped context.

Cheers, 

Mark

[Emily Jiang]

In the past, we discussed data access based on JPA but it did not get much support. I would like a data access model that can abstract away underline data structure, sitting on top of RDB or NoSQL. I think MicroProfile community showed great interest in CQRS and Event Sourcing. Can we somehow connect all dots to seriously look at how to deal with data?

My 2cents

Emily

[Andy Guibert]

Hi Emily,

A few months ago we did propose a template-based data access API that could sit on top of JPA and/or JNoSQL, but there were a few issues with that concept. We could have some basic operations to tie everything together, but it would have to cater to the least-common-denominator. This would result in most applications needed to access the lower-level technology (e.g. JPA or JNoSQL) anyway.

Regarding CQRS and Event sourcing... I listened in on most of the calls where Reza presented AxonIQ's CQRS/ES model, and even though I have a lot of experience in Java-based data access, it was a lot to digest. Also, it seems that CQRS/ES is a situational solution.  I think if we design an API that does CQRS/ES + Reactive all at once, it will be too much for devs to swallow and nobody will adopt it. That said, I'm sure whatever we come up with will still allow the CQRS model to be used, it just won't be a first class citizen of the API I'm proposing.

With this proposal I want to keep it focused on "accessing traditional relational DBs in a reactive way". We have JNoSQL leading the charge in the NoSQL world, but not much in the SQL space at the moment. The big RDBs like DB2, Oracle, and SQLServer are all extremely stable, have large customer bases, and many customers have their data already sitting with RDBs. I think a reactive API for RDBs (like R2DBC) has value because it allows customers to tap into their existing data in a reactive way.

[Andy Guibert]

Hi Mark, 

Yes, this is exactly what I had in mind! 

For those in the MP community that may not be very familiar with R2DBC yet, let me lay out the various pieces...

1) R2DBC SPI

    - lowest-level API, think JDBC level

    - this is the thing that DB vendors implement

    - only has a dependency on org.reactive-streams, but SPI impls need to choose some org.r-s impl (e.g. Reactor, RxJava, Akka Streams, MP Reactive Streams, etc)

2) R2DBC Client

    - very thin "convenience layer" around R2DBC-SPI

    - tied to Reactor

3) Spring Data R2DBC

    - not sure if it sits on top of R2DBC SPI or Client, but it could probably be based on either

    - tied to Reactor

    - provides the interface-based templates

What I was hoping to see come out of this proposal is a new form of an "R2DBC Client" that is based on MP Reactive Streams. Again, this can be a very thin layer so it should not be too difficult to implement.

As a bonus or possible next step, I think a repository-based solution would be very interesting. Something similar to Spring Data JPA, which allows the user to write their own operations on an interface, or extend a convenience template that provides basic CRUD operations.

[Werner Keil]

There's already a Template based API in Jakarta EE NoSQL.

We were asked about synergies with JPA when Otavio, myself and several others in this thread (Ondro, Reza, Emily) had a Jakarta EE panel discussion in Sofia at Java2Days. And this (Jakarta EE) looks like a better place and NoQSL is already in creation as first new Jakarta EE spec.

Some "Reactive" or CQRS layer on top which works against relational and non-relational databases, that is something more along the lines of MicroProfile IMO.

Werner

[Oliver Drotbohm]

I'd recommend to keep this discussion focussed on the reactive side of things as what builds on top of it is a different topic.

> Some "Reactive" or CQRS layer on top which works against relational and non-relational databases

That, however, is a non-starter. There's just no way to put a "reactive layer on top" as the underlying APIs *need* to be non-blocking in the first place and provide support for back-pressure. That's the sole reason we started R2DBC in the first place: neither JDBC nor ADBA expose those traits.

Cheers,
Ollie

> --
> You received this message because you are subscribed to a topic in the Google Groups "Eclipse MicroProfile" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/microprofile/Em24P1UCcPE/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to microprofile...@googlegroups.com.
> To post to this group, send email to microp...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/9300251a-8fee-4634-99ce-b991df153b45%40googlegroups.com.

[Werner Keil]

Yes, it should not duplicate anything Jakarta EE NoSQL or Spring Data already do ;-)

[Gordon Hutchison]

Hi Ollie,

Thanks for your interesting note.

Genuinely, I do find contexts very interesting, I should be honest and say we were aware of the context/transaction stuff and of Spring's reactive context right from

day zero when we discussed creating a MicroProfile Reactive Streams spec. (Ironically, I actually have a patent from 1997

that was all about this very area of having a transaction context cargoed about when we are hopping about threads and then reapplied

to the DB api use context thread https://patents.google.com/patent/US5893912 but it is now safely expired :-)

Transaction propogation in the precense of asynch, lock retention, 'workflow', check/savepoints/sagas etc. etc, all gets very complex.

My instinct is to try to root it in real world customer use cases or value add as there has to be compromises chosen.

For example, if R2DBC adds back-pressure to ADBA's purely asynch behaviour - what does this really mean in concrete terms:

1) when a user is reading a 'set' of things this has traditionally be done in terms of 'pulling' in batches at a speed driven by the client software plus an application cursor,

but between (fixed) batches the database is not working - what does it mean to change to dynamic request(N) and slow the database down using back pressure if that database

and its connections are shared?

2) when a user is writing a set of things too quickly for the database to keep up - the stream handling is already asynch before the DB client call so do a subset that repesents the overflow of the request(N) tickets just queue/block on submitting the write?

It is all very fascinating stuff were we can add lots of value by solving some very interesting problems :-)

Gordon.

Thanks for you note

[Gordon Hutchison]

Firstly, I must apologise for my typos above, I 'posted' in haste.

Question that have been bubbling in my head since though is:

   If we are attempting to do an update asynchronously but inside a transaction

   what do we do if that transaction ends at just the wrong time? For example

   'just' prior to the update being processed asynchronously?

   If we register the asynch work on initiation as being part of the transaction,

   what approach do we take to prevent locks being held for 'too long' if asynch

   means some arbitrary time later'?

   If request(N) ticketing prevents (db) server overload, how to we integrate

   insight of this with scaling-out, if that has traditionly been stimulated by slow-responses,

   real memory or CPU stress?

Gordon

Gordon

[Mark Paluch]

Hi Gordon, 

Thanks a lot for your questions and especially this type of questions. Let me try to come up with answers:

1) when a user is reading a 'set' of things this has traditionally be done in terms of 'pulling' in batches at a speed driven by the client software plus an application cursor,

but between (fixed) batches the database is not working - what does it mean to change to dynamic request(N) and slow the database down using back pressure if that database

and its connections are shared?

The underlying assumption is that we share a connection. This will not work as most database protocols are request/response-oriented without allowing to multiplex requests. 

From our experience with Postgres and SQL Server we learned that it makes sense to translate backpressure (for reading data) into a cursor-oriented thinking. This gives us the ability to fetch a chunk of data that does not overwhelm the consumer, pre-buffer a chunk of data to eliminate frequent roundtrips and we do not create too much overhead upon cancellation so we can just stop consuming the cursor. Reactive gives us some additional knowledge over the real demand and whether we satisfied the demand. This information allows us to fetch additional data while we emit data from the previous cursor read and this reduces overall latency.

2) when a user is writing a set of things too quickly for the database to keep up - the stream handling is already asynch before the DB client call so do a subset that repesents the overflow of the request(N) tickets just queue/block on submitting the write?

This is a bigger challenge to solve. In imperative/synchronous programming we have threads that protected the client and the database from overwhelming the database. If we run out of threads, no additional requests are issued. 

With reactive approaches, even just non-blocking, things are different. We can easily run out of file descriptors as we no longer need to wait before we can open another connection. 

I think we need some sort of semaphore/rate limiting that will provide us with a similar behavior we experienced with a threaded approach. I also think, that this kind of problem can be addressed by a connection pool capped at a particular size. Such a pool gets us to get more control over concurrency. 

Databases that support multiplexing at connection level (using MongoDB as example lacking a better, relational variant) typically have rate limiting built-in into their client/driver to prevent overflow to happen inside the database.

Regarding your additional questions, let me state that relational databases are not the best fit for reactive programming. They typically do not scale well (in comparison to NoSQL databases) and their intrinsic properties give us a natural limitation. 

With the move from imperative to reactive we move congestion out of the JVM and by using relational databases we move the congestion into the database itself. While reading use-cases are a well-suited use-case, modifying actions will probably suffer from ACID properties. That's not really different to what we see with imperative programming, however reactive programming aims for more efficiency and transactional locks do not play well.

   If we are attempting to do an update asynchronously but inside a transaction

   what do we do if that transaction ends at just the wrong time? For example

   'just' prior to the update being processed asynchronously?

Any asynchronous caller is responsible to clean up the transaction after it has received a completion signal of its work. Closing a transaction while async work is in progress will result in a failure. We should not assume sharing a transaction amongst multiple threads in the sense of concurrent workers. In a transaction, that is operated by reactive code we need safepoints that start a transaction, do work and another safepoint to cleanup the transaction.

   If we register the asynch work on initiation as being part of the transaction,

   what approach do we take to prevent locks being held for 'too long' if asynch

   means some arbitrary time later'?

This exactly touches my concerns about suitability for async/reactive patterns. Databases are a gatekeeper for throughput and they perform all the synchronization. Work piles up in the database and async callers get queued up. All progress is blocked by the database - or rephrase it: All work waits for the one async worker to complete that owns the transaction. Certain assumptions apply here such as all workers wait for the same lock and they to not spread work across multiple lockable entities.

   If request(N) ticketing prevents (db) server overload, how to we integrate

   insight of this with scaling-out, if that has traditionly been stimulated by slow-responses,

   real memory or CPU stress?

I don't have a real answer here. I feel that this boils down to stress/capacity testing to figure out the right settings.

Overall, reactive does not really change characteristics or performance of a database. We will still stick to reasonable connection pool sizes and decide within a connection pool how to deal with overflow work (reject/queue). We also do not expect database vendors to change their protocol/characteristics while it could make sense to include backpressure into the wire protocol.

Cheers, 

Mark

--
You received this message because you are subscribed to a topic in the Google Groups "Eclipse MicroProfile" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/microprofile/Em24P1UCcPE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to microprofile...@googlegroups.com.
To post to this group, send email to microp...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/01538810-f298-4285-8692-77b2590302fa%40googlegroups.com.

[Gordon Hutchison]

Thanks for your response Mark, I am still digesting chunks of it, but one thing I thought I would contribute is that when you said:

>>>With reactive approaches, even just non-blocking, things are different.

>>>We can easily run out of file descriptors as we no longer need to wait

>>>before we can open another connection.

It reminded me of the analogous problem and solution that

RSocket https://github.com/rsocket/rsocket/blob/master/README.md

has come up with whereby (made up terminology perhaps) reactive deals

with the vertical flow control of existing clients but does not handle horizontal

fan-in/scaling control of additional clients.

RSocket has the concept of 'leases'

https://github.com/rsocket/rsocket/blob/master/Protocol.md#lease-semantics

- maybe some of the patterns of that are applicable in this context.

:-)

Gordon

[Werner Keil]

So what is the exact goal?

The scope reduction to RDBMS puts it more along the lines of what a future Jakarta JPA spec might want to do, if the problem exists mainly with relational databases, and knowing, e.g. Jakarta EE NoSQL also has some ideas for Reactive.

If this is mainly intended as a wrapper around RSocket, what about other libraries?

Werner

[Gordon Hutchison]

Werner, this proposal will probably have little to do with anything in RSocket.

I merely referenced RSocket's 'lease' construct as it addressed the issue of trying to do flow(ish) control

on the number of clients accessing a service, to augment the reactive interface's request(N) construct which addresses

flow control from a single client.

Apologies if this muddied the water, it was probably an unnecessary distraction. Gordon.

[Andy Guibert]

Hi Werner,

I think that rolling reactive RDB functionality into JPA would be a mistake, as it would add considerable size to the already massive JPA spec. Since a reactive RDB API cannot be based upon JDBC, there would have to be a significant redesign/branching of the JPA spec/impl to account for using either JDBC or R2DBC at a low level.

Additionally, I think MicroProfile is the right place for this technology to brew because MP already has its own set of reactive APIs to integrate with. I don't think we would be allowed to have a JakartaEE spec depend on a MP spec. Also, since R2DBC is still experimental/incubating, MP seems like a more forgiving/lightweight place to prototype and iterate on this idea.

[Andy Guibert]

As for goals, I can see two possible technologies stemming out of this proposal:

1) A small/lightweight R2DBC + MP Reactive Streams client API. Basically, something along the lines of r2dbc-client, except instead of being tied to Reactor, it will use MP Reactive Streams. 

2) A template-based solution (similar to Jakarta EE NoSQL or Spring Data) that leverages CDI and MP Config based configuration, and allowing for JSON-B-style of marshaling/unmarshaling. Here I was thinking we could go for a trimmed down version of alternative technologies, with only basic operations/types built-in, and handling for more advanced operations/types handled by user-defined native SQL queries and adapters.

Since (1) would be a smaller effort I think we should start there, unless there is more interest in having (2) first. I have been prototyping (1) a bit, and would be happy to push what I have to the MP sandbox if others are interested in the concept and we think it's something MP may want to pursue?

On Thursday, January 17, 2019 at 6:13:19 AM UTC-7, Werner Keil wrote:

[Stéphane Épardaud]

On Friday, 11 January 2019 11:43:11 UTC+1, Oliver Drotbohm wrote:

Reactor is currently the only reactive library that exposes a Reactive Context (think of it as a ThreadLocal in the reactive world) that allows us to implement things like transactions. I.e. until other reactive libraries or APIs expose that kind of context, transactions will be hard (if not impossible) to implement and relational data access doesn't make too much sense without those.

Actually, MP-Concurrency provides precisely this, but more flexible than what Reactor does because it can be automatically propagated. Although for some reason, transactions is the only context we do not propagate by default. 

[Werner Keil]

MP-Concurrency? I thought there's also MP-Reactive, but there are so many features and proposals, it's hard to keep track of all them.

Among the reasons why claiming compatibility is virtually impossible to proof, and making some of the extensions (MP-Concurrency looks like yet another perfect candidate to merge into Jakarta EE Concurrency Utilities ;-) and wrappers available under Jakarta EE once they are mature enough.

[John Clingan]

+1. Keep them separate. Anyone remember the EJB spec? ;-)

[Oliver Drotbohm]

> Am 21.01.2019 um 15:04 schrieb Stéphane Épardaud <stephane...@gmail.com>:
>
> Actually, MP-Concurrency provides precisely this, but more flexible than what Reactor does because it can be automatically propagated.

Can you point me to anything reactive in MP-Concurrency? All I can find are references to CompletableFuture.

Cheers,
Ollie

[Werner Keil]

I'm not talking about JPA at least not in the first place, this MP Concurrency feature or the JAX-RS Client are certainly candidates for better alignment with Jakarta EE.

The reactive stuff, sure, if this is as experimental and unstable as e.g. the various drafts of JSON Schema, keep it here till the underlying standard has matured, I don't think I would like to see JSON Schema in either JSON-P or JSON-B specs at Jakarta EE yet ;-)

[Arjan Tijms]

Hi,

On Monday, January 21, 2019 at 9:53:54 PM UTC+1, John Clingan wrote:

+1. Keep them separate. Anyone remember the EJB spec? ;-)

On the other hand, there's also been JSF Managed Beans, JAX-RS's own managed beans, CDI beans, Servlet components, etc.

Creating totally or largely independent things all the time does not necessarily make the platform more consistent or easier to use either.

Kind regards,

Arjan

[Andy Guibert]

Hi Arjan,

> On the other hand, there's also been JSF Managed Beans, JAX-RS's own managed beans, CDI beans, Servlet components, etc.

Generally I agree with you here, but it's important to note that CDI was designed after JSF/JAX-RS/Servlet managed objects, in a way where explicit attention was paid from day 1 on keeping the CDI spec decoupled from any underlying technology. JPA on the other hand, was not designed to ever be decoupled from JDBC, and trying to shoehorn in a JDBC alternative after JPA is well established seems very disruptive to the existing JPA users and implementors.

> Creating totally or largely independent things all the time does not necessarily make the platform more consistent or easier to use either.

One goal of this proposal is precisely to make the platform (MicroProfile) more consistent and easier to use, by having first-class support for MP Reactive Streams when accessing relational DBs in a reactive way. Also, by keeping this separate from JPA we will have the freedom to succeed/fail/iterate independently of JPA.

For starters I think a small/simple low-level API (think JDBC) would be an appropriate way to get the ball rolling with reactive data access. I think it's important to focus on the low-level aspects of this relatively new technology before we look to build on top of it or integrate it with more mature technologies (like JPA).

[Arjan Tijms]

Hi Andy,

On Friday, January 25, 2019 at 9:31:29 PM UTC+1, Andy Guibert wrote:

Generally I agree with you here, but it's important to note that CDI was designed after JSF/JAX-RS/Servlet managed objects, in a way where explicit attention was paid from day 1 on keeping the CDI spec decoupled from any underlying technology.

I don't entirely disagree with you here, but it was a little less rosey than that in actuality. CDI grew out of the WebBeans proposal, which was initially conceived to bridge JSF and EJB. JAX-RS is the historic mistake where the JAX-RS group for reasons that are lost in time (I tried to dig a little, but couldn't find much) felt it necessary to invent their own DI and component model. When this historic mistake started to dawn upon the people involved it was already too late, and only in one of the last revisions before CDI 1.0 went final some adhoc bandage was added to make the two new technologies somewhat compatible.

In the beginning especially not everyone understood CDI being a foundational technology and CDI-as-better-EJB camp was still there, hence CDI having producers for things like the HttpServletRequest (making it dependent on Servlet).

 

JPA on the other hand, was not designed to ever be decoupled from JDBC, and trying to shoehorn in a JDBC alternative after JPA is well established seems very disruptive to the existing JPA users and implementors.

I'm not 100% sure of this.  Look at https://objectdb.com for example, which is not a JDBC compatible database or http://www.datanucleus.org/products/accessplatform/jpa/getting_started.html

So while it's true that JPA was designed with relational databases in mind, the core object modelling aspects can be used for storing into other kind of data stores, and extensions have proven it can indeed be done.

Naturally, it wouldn't always make sense to transparently retarget JPA based code from relational to non-relational, but IMHO it does make sense to re-use core parts of JPA for various persistence solutions.

The way things are going now we'll end up with JPA some totally different MP Persistence API, and some totally different Jakarta EE NoSQL API, and of course there will still be JDBC.

Within a given application server such as Payara or Wildly we'll thus end up with basically 3 totally different APIs, each with their own packages, support code, utilities and what have you.

I'm not sure if that's the way forward.

 

 by keeping this separate from JPA we will have the freedom to succeed/fail/iterate independently of JPA.

Unfortunately that's the argument how most NIH projects start. IMHO again, you can be just as independent when and iterate just as much when building on top of JPA to start with.

Kind regards,

Arjan

[James Roper]

On Thu, 10 Jan 2019 at 19:05, Ondro Mihályi <ondrej....@gmail.com> wrote:

This is a great idea and nicely fits into existing reactive efforts (operators and messaging).

From what I know about R2DBC and ADBA is that R2DBC is the only fully reactive API, with support for backpressure. ADBA is only asynchronous, so it's useful, but not complete as it doesn§t support flow control.

This is not true. R2DBC uses Reactive Streams for all operations, even operations that aren't streams (ie, they return a single value). ADBA uses CompletionStage for operations that return a single value, and Reactive Streams for operations that return streams of values. Both of these approaches inherently support flow control, Reactive Streams explicitly through signalling demand, CompletionStage implicitly through controlling the redemption of CompletionStages that you supply. Both approaches are fully reactive, it's just that where it makes sense, ADBA uses CompletionStage for single values rather than forcing everything to be treated as if it were a stream, which in my opinion is a better approach because CompletionStage is very lightweight and provides a lot of useful features specific to working with just single values, additionally it integrates better with the rest of the Java ecosystem that uses CompletionStage, such as JAX RS, the JDK itself (including the new HTTP client in the JDK) etc.

 

That's basically the reason why Pivotal started R2DBC. Another drawback of ADBA is not just that it's a JDK effort and will require a certain version of JDK but that it's been talked about for years and there's still nothing available.

I'm all in to explore R2DBC and even cooperate with Pitoval folks on its development while working on an extension or a spec to integrate it well into MP.

Ondro

On Wednesday, January 9, 2019 at 3:56:10 PM UTC+1, Andy Guibert wrote:

Hi all,

Recently there have been efforts within MP to define a standard set of reactive APIs, namely MP Reactive Streams. When considering relational DB access, there are a number of solid options today, such as JPA, Hibernate, MyBatis, or jOOQ. However, all of these options are based on JDBC, which is a blocking API.

When we look at _reactive_ relational DB access, there are two main efforts brewing:

• R2DBC (Reactive Relational DB Connectivity, open source, driven by Pivotal)
• ADBA (Async DB Access, internally designed, driven by Oracle)

Neither of these efforts are officially supported yet, but R2DBC looks like it's close. Additionally, it has the advantage over ADBA that it's a library, as opposed to ADBA which is proposed for inclusion in the JDK (and won't be included until JDK 13 at the absolute earliest).

I think it would be interesting to experiment with combinations of R2DBC and MP Reactive Streams to see if we can come up with an API that allows applications to access relational DBs in a simple, extensible, and reactive way.

Anyone else interested in collaborating on this, or have further thoughts/questions?

Thanks,

Andy Guibert

--
You received this message because you are subscribed to the Google Groups "Eclipse MicroProfile" group.
To unsubscribe from this group and stop receiving emails from it, send an email to microprofile...@googlegroups.com.

To post to this group, send email to microp...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/7edb2a0b-f6ee-46fe-a88b-b5774bfbc001%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

James Roper
Architect, Office of the CTO, Lightbend, Inc.
@jroper

[James Roper]

I think Emily's point is good.

It's important to delineate the difference between reactive programming and reactive systems. When we first started MP Reactive, we created a document that outlines this, which is required reading before you read the rest of this email:

https://github.com/eclipse/microprofile-reactive-streams-operators/blob/master/approach.asciidoc

So, things like MP Reactive Streams, R2DBC, Reactor etc, these are all reactive programming technologies. Now reactive programming by itself has its value - it allows more efficient use of resources on a single machine. But the real value in reactive is in reactive systems, which requires reactive programming, but is much bigger than reactive programming. Reactive systems, ie, systems architected using reactive principles, are able to deal with the uncertainty of cloud native, where things crash all the time, where you can't rely on ACID transactions to ensure consistency because each service has its own database, where one minute a service may be served by 3 nodes, then 100 nodes, and then back down to 3 nodes again. Your traditional n-tier architectural patterns fall flat on their face in this environment, because you're now programming a distributed system. The thing you don't want to do is have to deal with failure, because that's messy and error prone, you want some pattern or technology that you're using to deal with it for you so you can just take consistency, resilience, etc for granted. In a monolith, ACID transactions gave you that. In the cloud, you don't have that, and so the patterns that depended upon ACID transactions don't work, instead you require completely new patterns, patterns for updating state, for propagating those updates through your system, for ensuring services are in sync, that allow you as the programmer to not have to deal with failure, so you can take consistency, resilience and scalability for granted. Reactive patterns like CQRS, messaging, and event sourcing give you this, they can guarantee causal consistency across your system at scale, propagate updates throughout your system, will handle failure for you, etc etc.

This is where relational databases and the libraries on top (eg JPA) have limited applicability (not no applicability, just limited), because they too are designed around the old patterns. So using reactive programming to access the database is interesting, but perhaps not interesting enough to warrant all the work to, for example, create a reactive version of JPA, and perhaps not interesting enough for community to adopt it. At least in the foreseeable future, it's probably better to stick with blocking IO there. That doesn't mean that efforts like R2DBC and ADBA aren't useful, it just changes who they are useful too - they are useful to the implementors of Event Sourcing and CQRS libraries, as they allow these libraries to make efficient use of resources on top of a relational database, as they implement the reactive systems patterns that they offer.

So I'd be more inclined to focus on the high level reactive systems features, like messaging, like CQRS, like event sourcing, for user facing reactive APIs in MicroProfile.

Regards,

James

--

You received this message because you are subscribed to the Google Groups "Eclipse MicroProfile" group.
To unsubscribe from this group and stop receiving emails from it, send an email to microprofile...@googlegroups.com.
To post to this group, send email to microp...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/5b873665-1f69-4c41-b5d9-5501c510cfc1%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Rhuan Henrique]

Hi,

I think Jakarta Community will improve JPA in future and will promote support to reactive programming and CompletionStage. Then, I think Microprofile Community don't need care about that now. Furthermore, the Jakarta EE will promote support to NoSQL with Jakarta NoSQL, and the Microprofile. Microprofile is a project to extend the Java EE/Jakarta EE feature to attend to microservice architecture, then we shouldn't promote solutions that Jakarta EE will promote.

Thank you

Rhuan Rocha

SouJava Member 

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/CABY0rKMk8e_uz-LNR7qWFAmkro0mLk8m8baDuZqFsaSQBwJvew%40mail.gmail.com.

[Andy Guibert]

> R2DBC uses Reactive Streams for all operations, even operations that aren't streams (ie, they return a single value). ADBA uses CompletionStage for operations that return a single value, and Reactive Streams for operations that return streams of values. ... Both approaches are fully reactive, it's just that where it makes sense, ADBA uses CompletionStage for single values rather than ...

The primary reason I prefer R2DBC over ADBA is that R2DBC is a library, whereas ADBA is tied to the JDK, meaning nobody can use it in production until JDK 17 at best. IMO that's way too long of a wait. If ADBA were to decouple itself from the JDK, I would more closely consider the pros and cons between the two.

Also, it's important to note that R2DBC at its core is just an SPI. I am proposing we build a MicroProfile RS-centered API on top of this SPI. The API can double as an adapter to correct some of the design quirks like returning a stream for single-value operations. In fact, in my current prototype I actually have the API returning CompletionStage for single-value operations.

> So I'd be more inclined to focus on the high level reactive systems features, like messaging, like CQRS, like event sourcing, for user facing reactive APIs in MicroProfile.

I agree that MicroProfile should pursue all of these items, however, my concern is that CQRS/ES seems like a fairly situational data access strategy. Also, going from current data access technologies to reactive CQRS/ES requires users to take quite a leap in their understanding of reactive programming. Today we have JDBC and JPA for example -- some people still use JDBC directly because of its simplicity and how general purpose it is. Just like JDBC was an important building block for JPA, I see this potential MP R2DBC API as an important stepping-stone/building-block towards other solutions (CQRS, ES, reactive JPA, etc).

- Andy

[Gordon Hutchison]

I am fairly sure in my gut that leaving this to happen someday in a future Jakarta/JPA spec is not the best move.

One thing to state up front is we have had a good example in doing great work in MicroProfile Config and for that to go on and

be repurposed in a JSR.

However the main thing is this, although flow control is useful at a systems perspective, I have never seen a customer situation where the

problem was flow control between a trad database client code and the database engine. Think it through - for reads, users

pull at cursors which are paged and for writes the databases just get slower causing the problem to back up into the writer.

So it is the INTEGRATION of the data layer into streams that is the value add at this level (i.e. not the reactive systems level

but the programming model level).

What opened my eyes to this lesson was 58:24 to 58:44 at https://www.youtube.com/watch?v=-ioxYn9Vlao

You only get that insight when you code, or are exposed to the density of the reactive stream based composition code, or if a coder tells you.

IMHO. 

So it is if one integrates with something like Reactor/Webflux stack or MP Reactive Streams (once we have populated it fully) across other technologies that this benefit emerges -

not one spec at a time, doing back-pressure, but a bit differently to the code on either side for anything above basic org.reactive streams.

Gordon.

[Oliver Drotbohm]

> Am 30.01.2019 um 08:09 schrieb Andy Guibert <andy.g...@gmail.com>:
>
> Also, it's important to note that R2DBC at its core is just an SPI. I am proposing we build a MicroProfile RS-centered API on top of this SPI. The API can double as an adapter to correct some of the design quirks like returning a stream for single-value operations. In fact, in my current prototype I actually have the API returning CompletionStage for single-value operations.

Just to get this straight: using reactive streams for both single-value and collection-value operations in R2DBCS is not an oversight. It's a deliberate decision to semi-unify the programming models for both as in more functional world treating them as fundamentally different is a bug, not a feature.

In reactive pipelines single values often explode into a stream ("Find customers by name like"), get reduced into a single value and exploded again. Imagine that in each of those steps the programming model changes fundamentally. Reactive pipelines usually quite a bit of effort in debugging and test. Why would you want to complicate this even further by switching programming models right within the pipeline definition?

Reactor has dedicated types for collection-value and item-value types (Flux / Mono) but their APIs and the fundamental programming model are the same in both. That consistency has been a key aspect in attracting developers in projects we see Reactor and Spring WebFlux in use.

> > So I'd be more inclined to focus on the high level reactive systems features, like messaging, like CQRS, like event sourcing, for user facing reactive APIs in MicroProfile.
>
> I agree that MicroProfile should pursue all of these items, however, my concern is that CQRS/ES seems like a fairly situational data access strategy. Also, going from current data access technologies to reactive CQRS/ES requires users to take quite a leap in their understanding of reactive programming. Today we have JDBC and JPA for example -- some people still use JDBC directly because of its simplicity and how general purpose it is. Just like JDBC was an important building block for JPA, I see this potential MP R2DBC API as an important stepping-stone/building-block towards other solutions (CQRS, ES, reactive JPA, etc).

I've seen this coming up in this thread: there is no point in trying to put some reactive lipstick on a blocking API. This is usually done via some async (read: thread pool based) intermediate. What I think is dangerous about this (and one of the primary reasons we don't support reactive types with JPA and JDBC repositories in Spring Data) is that it creates the impression of something happening that is not happening. You'll totally wreck the benefits of the reactive approach and still pay the price of more complex debugging and testing. What's the point?

This also applies to trying to use JPA for NoSQL stores. Users end up with code that's guaranteed to only half work (@Column doesn't make too much sense on any of NoSQL stores, what about transactions on non-transactional stores?). Slightly related: trying to unify all document stores under a single document store API. It's a bad idea as the devil is in the details and NoSQL stores are – much more than relational databases – chosen because of an individual store's capabilities. Try to put a canonical indexing API around both MongoDB and Couchbase. I guess you can, it's just gonna lack all the things a user of the two stores is going to look for. IMO, trying to put an API on a technology space that's defined by what it is *not* – and by that of course is incredibly diverse –, is just the wrong premise in the first place.

Even asynchronous JDBC is a straw man really. Which queries would you want to execute asynchronously? Long running ones, right? The longer the operation is running, the more likely it is to cause contention on the pool and you either end up with huge pools or calls being blocked waiting for a connection from the pool. Again, the bottle neck is resource consumption in the JVM with that. In a way, reactive database access is pushing the resource bottleneck down the stack – from the JVM down to the database who's historically much better dealt with high-concurren

Reading the above paragraphs again, I realize that I basically gave a rationale of why the Spring Data portfolio looks as it looks :). For us the key really is to unify via the programming model (repositories, templates, mapping annotations) and expose store specifics in the APIs.

Cheers,
Ollie

[Gordon Hutchison]

Re: waiting for JPA to do it:

We need to fill out the MicroProfile Reactive stack in order to get the benefits of users

being able to compose micro services using the functional composition of reactive

streams across different technologies. Think of app code like:

```

source.via(appsChecker).standardOperator(appsLambda).to(appsStore).run();

```

Without this ability a java enterprise developer has to learn three or four APIs in order

to get things in an out of local java variables to go from one stage/spec to

another down a chain of processing stages. With a common approach to reactive

streams across components this level of detailed API knowledge is not needed at

the connecting code.

The approach of MicroProfile Reactive Streams is to build a data structure (graph)

using a fluent builder API which is then passed to the infrastructure to 'run'.

This allows platform vendors to insert 'value add' stuff into the stream's graph

prior to running it. So the users bits are 'lifted' into the final stream that is 'run'.

There is a lot of benefit in each 'bit' being a similar type rather than a hodge-podge

of Reactor/RxJava/Lightbend/Helidon/RawReactiveStreams all thunked to the lowest common

denomenator of Publisher-Subscriber.

If we don't do any work to create MicroProfile Reactive Streams integration of different

technologiues and leave it to individual legacy specs to sprinkle in something reactive...

o where are the 'operators' going to come from? (massive of the shelf value add)?

o will application code be expected to write raw Publishers/Subscribers? (nuts)

o if each (e.g.JPA) invent something (new) at a level higher than org reactive streams will they coordinate with

   the other specs?

o will the new version of their spec be backwards compatable and contain the rest of the pending wishlist?

o if so how long will it take to get the value into customer's hands to play with?

I remember in the past there used to be one, carefully stewarded,

way to do each thing in enterprise Java. Those days are long gone.

We will 'suceed with speed' or we will be out evolved by

propriatory and de-facto standards which will either be single vendor

driven or harder to integrate.

There is certainly customer value in this area, regardless of which standards body

creates it. Most MicroProfile platforms are faceted anyway, so

if we do produce this spec standard and users decide not to use it, natural selection

will occur and not much harm will be done. The worst that would happen is that there

would be a bottom bar the JPA equivalent version would have to be better

than and they (and we) would get a head start in understanding what works.

Ship early, ship often, go MicroProfile.

Gordon

On Thursday, 17 January 2019 13:13:19 UTC, Werner Keil wrote:

[Gordon Hutchison]

I think we do need to search for how we can add value for users

as they move to new architectural patterns like:

   https://martinfowler.com/eaaDev/EventSourcing.html

   https://martinfowler.com/bliki/CQRS.html

   asynchronous connection of microservices using Kafka etc.

but we can't bite off and chew all those in one piece of work

and to do them seperately means that they will come down

to the existing database APIs and patterns when the data comes

to rest.

If each of these areas of concern wants to be non-blocking, error

handling, flow-controlling, context propogating, hetero-traced etc.

it seems inefficient to me to have them all do this without any

common foundations. Particularly as the different back-ends have

different APIs at the moment that deal with these areas of concern.

Would it no be wise to have a common model for a reactive stream

source/sync that is the data at rest which handles these concerns?

Higher level structural/architecture value add can be composed ontop

without having to deal with the 'reactive' concerns other than as a

common plug-point.

What is the value add of slick, e.g as expressed here:

   http://slick.lightbend.com/talks/scaladays2015sf/Reactive_Slick_for_Database_Programming.pdf

In the scala/lightbend world would one not build on top of slick?

For example, at

   https://blog.codecentric.de/en/2017/02/cqrs-event-sourcing-lagom/

   https://www.lagomframework.com/documentation/1.2.x/java/PersistentEntityRDBMS.html

I see components building on top of reactive database client layers, not the

need for those higher layers meaning it was not a good idea to build the lower ones,

for example Slick?

Don't we need to build these things in layers in order to chunk up the complexity,

few individuals have knowledge/experience at the combined multiple-levels to

be able to solve it all at once, let alone take a community with them.

In MicroProfile, for building a reactive stack, I think we need to chase

the tail-lights/experience of single vendor solutions like akka,

and webflux (and soon helidon) and build the stack in layers.

Otherwise we will have a wide front and make slower progress.

Gordon

[Gordon Hutchison]

I think we need to be clear that we (I think) have a concensus that we are only talking about using the R2DBC SPI

not the upper layers that use Mono.

I do agree with what James says about CompletionStage etc, but wanted to add the above to:

 "R2DBC uses Reactive Streams for all operations, even operations that aren't streams (ie, they return a single value)."

I don't think the detailed backend SPI is worth having a long debate over though as a driver that

supports what we are talking about here will just have to use the 'best' SPI that is available for the

particular store version that the user wants to use. The whole point is that they are unlikely to even

have to concern themselves much about the particular SPI at all. The layers above will just have

to work with what they can to support the features that are feasible.

In the future, any MP Reactive Data Client code is likely to exist to plug into more than one

Database SPI. We just need to get on and work with what we can use today and adapt

and add on more SPI driving capability as they appear and become useful.

'Just say Mon no'

Gordon.

[Gordon Hutchison]

I just wanted to chip in that I that I did not want my sign off  'just say mon no'

to be taken as anything other than light hearted - and I did not mean for

it to be the last line in this thread :-)

When I first came across the concept of Flux and Mono both having

the same super class that one could then write operations/operators

against I do remember thinking it was a neat design choice.

Sometimes I wonder if as well as first mover advantage, arriving early

means things come and sit down bedside you later on which change the

scenery.

When Mono was written there was little use of CompletableFutures

and having an 'isa' relationship with org reactive streams' Publisher

for ones value add classes was pretty unquestionable if Flow did not

even exist let alone all the other advantages.

I do wonder if Flow will ever replace the org reactive stream classes from an end

user point of view and if this does happen what path for Flux/Mono -

are they are 'big' enough now that this is maybe a factor in Flow

not getting much energy for adoption?

Gordon