Architecture of v2 vs v4

[Ken Finnigan]

Hi Community,

Since the announcement a few weeks back of some PoC code that’s targeted to be v4 of Thorntail, there have been many questions around what that means for v2 and how they differ.

Firstly, the existing codebase from WildFly Swarm is being migrated to become Thorntail v2, with it's first release coming soon. It will continue to be developed and updated by the core team at Red Hat as well as by the community for some time.

With respect to v4, yes it is a different architecture to v2. The current code is built on top of WildFly subsystems and utilizes a modular classpath. The current architecture for v4 utilizes the upstream projects of WildFly directly, such as Undertow and RESTEasy, while operating with a flat instead of modular classpath. It also embeds Eclipse MicroProfile implementations more directly than v2 ever could. With the switch to a flat classpath, we also see a significant reduction in the dependencies that we need to develop for Thorntail v4, as it’s become infinitely easier to simply include a framework or library directly into a users application. Additionally, development can be accelerated under this new model. Further details on the v4 architecture will follow.

With v4 we will continue to focus on Eclipse MicroProfile, while additionally focusing on appropriate Cloud Native technologies for OpenShift/Kubernetes. Some of these key pieces are projects such as Weld, RESTEasy, Hibernate, and Undertow. Our goal is still to make it easy for developers to use their years of knowledge investment to walk forward into developing microservices with Enterprise Java.

As v4 is currently not the main focus of the team. If there are features and use cases from v2 that the community would like to see in v4, please let us know by creating an issue so everyone is able to assist and contribute. It can then be determined whether the use cases are appropriate for v4 or are a better fit in another project.

As always, any and all feedback is greatly appreciated.

Thanks

Ken

[Erik]

Thanks, Ken, for the clarification.  As a user (and fan) of SF Swarm, I'm glad to see the core goals are still part of v4. 

To be honest, one concern I've had with v2 is the memory footprint.  This adds to ongoing costs in a horizontally scaled Kubernetes cloud deployment. Yet, I don't want to lose features.  From my perspective, I'm using most core JEE features, though I recognize the need to clean up some older code using remote interfaces.  Unit testing is driving that, as I've just begun to do end-to-end unit testing from the REST to the DB using JPA. 

Can you put more color on your Kubernetes plans?  That is currently where I deploy to, GKE currently.  I don't have any issues with it there, except perhaps the cost of the large memory footprint.  So curious what additions you plan to add. 

One thing comes to mind is perhaps help with health checks.  Though I don't have any specific requirements there largely because I haven't gone beyond basic health checks.  It is also difficult for me to keep up on the new features in Kubernetes.  But, I imagine improvements both in Kubernetes and deployed services will likely be around better orchestration of scaling and health.  

One thing that might help us is documentation on what parts of V2 may not initially be supported in V4.  That would save us the time of having to test and raise issues on an after the fact basis, and raise concerns earlier about needed capability. 

Thanks for your commitment and openness to the community, and helping to create a great project that lets JEE shops quickly begin producing microservices without having to spend time identifying and working around non-JEE framework limitations.

[Jeff Zhang]

Hi Ken,

Thanks for information.

Does thorntail v4 have plan to support JDK JPMS? I read it from the document (https://docs.thorntail.io/4.0.0-SNAPSHOT/#flat-classpath) but is there any further discussing?

Regards
Jeff

[Ladislav Thon]

I can't authoritatively answer your questions, but my understanding is that v4 should be much better when it comes to both memory footprint and speed of tests. When it comes to K8s health checks, I think MicroProfile Health is what you're after, and there's no difference in that between v2 and v4.

LT

--
You received this message because you are subscribed to the Google Groups "Thorntail" group.
To unsubscribe from this group and stop receiving emails from it, send an email to thorntail+...@googlegroups.com.
To post to this group, send email to thor...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/thorntail/7b039190-224b-4cf5-b2e5-81e6921bdffa%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Werner Keil]

One would assume so, although JBoss had its own module system in the past. 

With new JSRs like 385 we also started Jigsaw support (but with Java 8 as a minimum baseline, where absolutely necessary we also consider the use of multi-release JARs) 

Regards,

Werner

[Ken Finnigan]

There's a few questions from different people so I'll answer in turn.

Erik,

v2 certainly has higher memory footprint requirements than v4, which is one of the reasons for the architecture change to significantly reduce that.

Our plans around Kubernetes/OpenShift are that what those environments provide, we won't replicate. The best example of that is in v2 we had the concept of "topology", with different implementations, which was our way of doing service discovery. In v4 we won't be providing anything like that as we will assume deployment to a Cloud, such as Kubernetes/OpenShift, where that functionality is provided by the platform.

Also, we would like to provide better integration with functionality from Kubernetes/OpenShift for things like ConfigMap. There's already an issue about being able to dynamically reload configuration when a change in the ConfigMap occurs.

We can certainly look to provide some details around what from v2 isn't likely to be in v4, such as topology. Would a document on the website work best?

Finally, thanks for being a user of the project! We love getting great feedback about how the project is used to know the right direction forward.

Jeff,

Yes there are plans to run on/support JPMS. Right now we need some of the upstream projects we depend on to add module support for JPMS before we can. That being said, over the coming months we're going to look at getting v4 running on JDK 10 with a view to having it ready for when JDK 11 LTS is released.

Thanks

Ken

[Erik]

Thanks, Ken!  You are right.  I abandoned the topology features as soon as I got into K8S and used their DNS based service lookup.  I love it.  Super simple.  No extra code.  Just need name of your service and K8S does the load balancing.  I use the WFS YAML to separate local from K8S development, where the only differences are the host name used for connecting to other tiers, such as DB, or Web to Microservice. 

Any document comparing V2 to V4, particularly in regard to functionality that will or won't be included in V4, would be a very helpful start. 

I'm looking forward to and am excited about the memory footprint and performance improvements in V4.  As long as core JEE functionality is preserved in V4, and YAML has same functionality, I don't think there will be any issues.  That's primarily my concern.  

Regarding the POM, will this be largely backwards compatible?  Will we be able to just pint to a new version, and watch it magically work if our dependencies are included in V4? 

Most of us are probably on JDK 8, self included.  Not sure what is involved with a move to JDK 10.  Hopefully, it just works. 

I plan to spend some time myself trying to review the Docker image to decrease the footprint.  Without looking (not on this computer), I think it currently runs on Ubuntu, which is a bit heavy, rather than Alpine, which I have discovered is a much smaller footprint.  It dramatically reduced my web/static tier when I switched to Alpine.  So, if we need to deploy JDK 10, the first question I am going to have is there an Alpine or similar low footprint image we can use. 

I had an issue upgrading Docker on Alpine, where I had to revert.  While that doesn't impact microservices (V2/V4) deployment, it makes me concerned that not every upgrade is guaranteed to go smoothly with it.  If V2 works with Alpine, but V4 only works with Ubuntu, that would negate the memory footprint benefits of V4. 

[Ken Finnigan]

Comments inline

On Tuesday, June 5, 2018 at 9:22:28 AM UTC-4, Erik wrote:

Thanks, Ken!  You are right.  I abandoned the topology features as soon as I got into K8S and used their DNS based service lookup.  I love it.  Super simple.  No extra code.  Just need name of your service and K8S does the load balancing.  I use the WFS YAML to separate local from K8S development, where the only differences are the host name used for connecting to other tiers, such as DB, or Web to Microservice. 

Any document comparing V2 to V4, particularly in regard to functionality that will or won't be included in V4, would be a very helpful start. 

I'll create an issue for us to write something.

 

I'm looking forward to and am excited about the memory footprint and performance improvements in V4.  As long as core JEE functionality is preserved in V4, and YAML has same functionality, I don't think there will be any issues.  That's primarily my concern.  

Depends what you mean by "core Java EE functionality". We're still going to offer most of the Web Profile, just not EJBs and likely not JSF. Anything above the Web Profile is currently not something we're going to add to v4.

As for the YAML, it's going to have a different structure, as we're no longer based on the management model from within WildFly. So in some respects it's "easier" because there are no where near as many configuration choices to be made, but they are going to be different.

 

Regarding the POM, will this be largely backwards compatible?  Will we be able to just pint to a new version, and watch it magically work if our dependencies are included in V4? 

There's a basic migration tool for pom.xml that has been developed already, but I don't think it's had much use yet to know whether there are any kinks or not. This tool also migrates the config files from v2 to v4, but it doesn't have a full set of conversions yet. More than happy to accept PRs to add more!

There will be some differences, as v4 artifacts are prefixed with "thorntail-".

 

Most of us are probably on JDK 8, self included.  Not sure what is involved with a move to JDK 10.  Hopefully, it just works. 

What we'd look to do is add support for JDK 10, but still have it runnable on JDK 8. So you wouldn't be forced to update to JDK 10 with v4 unless you wanted to.

[Ken Finnigan]

It's possible, but not something we've tried yet. There is a member of the community experimenting with GraalVM, but will have to see what they find.

[adding Thorntail group again]

Ken

On Tue, Jun 5, 2018 at 9:59 AM, Jianfeng Zhang <shi...@gmail.com> wrote:

Got it, thanks Ken.

BTW, I just see the article "Eclipse Vert.x goes native" (https://vertx.io/blog/eclipse-vert-x-goes-native/). Vertx could be natived by GraalVM. I think it's wonderful.
Do you think Thorntail (or wildfly-swarm) could go native too? Or someone had tested it and please share result information?

Thank in advance

Jeff

To unsubscribe from this group and stop receiving emails from it, send an email to thorntail+unsubscribe@googlegroups.com.

To post to this group, send email to thor...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/thorntail/6903130d-039e-46b2-847d-fe65956b05da%40googlegroups.com.

[Jeff Zhang]

Got it, thanks Ken.

BTW, I just see the article "Eclipse Vert.x goes native" (https://vertx.io/blog/eclipse-vert-x-goes-native/). Vertx could be natived by GraalVM. I think it's wonderful.
Do you think Thorntail (or wildfly-swarm) could go native too? Or someone had tested it and please share result information?

Thank in advance
Jeff

[Jianfeng Zhang]

Thanks Ken, Hope Thorntail will go native soon. :)

[Sorry for wrong thread reply]

Regards

Jeff

To view this discussion on the web visit https://groups.google.com/d/msgid/thorntail/CAKeeVe5aJh37cOg3mrTzgB9m6dewVvfE%3D-689QAr8ErL5Z2v%3Dg%40mail.gmail.com.

[Erik]

Ken, that is good news on the POM migration.  YAML changes sound like something we can easily work through.  TY for JDK 8 support.  The less day 1 change, the better. 

I could care less about JSF, JSP or any UI components other than REST.  The standard is pretty much REST and WebSockets for UI integration, with UI being a different layer (e.g., Angular). 

What do you mean by "just not EJBs"?  I use EJBs heavily.  Though, I'll strip out the remoting code if needed as this is dead code from 2002.  I only needed support to have it compile for now.  The only hard part about stripping it out is I don't have time to rewrite all its functionality using REST APIs.  But, the goal is REST/WebSockets only for client interfacing. 

A quick google suggests the web profile includes "EJB 3.1 Lite", JPA and JTA, all of which I use.  Simply because of unit testing, I have been migrating away from not only remote interfaces, but explicit local interfaces, in favor of @LocalBean.  And I use JPA. 

https://jaxenter.com/introducing-the-java-ee-web-profile-103275.html

To be sure, I can see uses cases that don't require these, as I have another microservice that uses CrateDB instead of MariaDB.  For that, I don't use JPA, but just JDBC.  It seemed like a sacrifice in the beginning to not have JPA.  But, when you are using a high performance well indexed document database with sharding, you discover that data-modeling, and thus the benefits of JPA, are far less important.  This changes how you view transactions as well, since a commit, instead of encompassing multiple tables, will often only touch one table, which contains a document with a lot of depth.  

I have discovered that for now at least, a hybrid approach using traditional SQL (and thus JPA/JTA/EJBs) for ACID along with a document database gives you the best of both worlds.  I've seen a lot of requirements also including both. 

My apps all depend on the EJB Timer service excluded from the Web Profile.  I'm not sure why anyone would want to remove timers.  But missing this could be a show stopper for upgrading to V4.  Example use case: User registers.  That can be synchronously saved in DB.  But, propagating to another API requires scheduling, as other API may not alway be up.  I use a timer to handle in-memory task queuing.  In short, my apps do a lot asynchronously to ensure API calls are not blocking and communication availability with external APIs has fault-tolerance. 

I have an app I haven't converted to microservice, yet, but plan to, that relies on MDBs that read a live in-memory queue and broadcast using Observables.  The queue is separately populated with live real-time data.  One impact is the UI is constantly updated via web sockets with live data that ultimately originates from an external streaming API, but transformed via the microservices app.  This requires queuing to handle the large quantity of real-time data in addition to the source of the data being properly loosely coupled from the web use case via the queues. 

Leaving out asynchronous or queuing is ignoring real-time applications and their requirements. These are not traditional monolithic use cases.  This is not tying into an enterprise bus.  It is leveraging modern web capabilities in a real-time world, connecting real-time APIs with web clients being a part of the channel in addition to automated clients (other consumers, service accouts), both for input and output, including the use of web sockets in addition to REST APIs. 

On Tuesday, June 5, 2018 at 9:39:08 AM UTC-4, Ken Finnigan wrote:

Comments inline

[Ken Finnigan]

On Tuesday, June 5, 2018 at 4:38:05 PM UTC-4, Erik wrote:

Ken, that is good news on the POM migration.  YAML changes sound like something we can easily work through.  TY for JDK 8 support.  The less day 1 change, the better. 

I could care less about JSF, JSP or any UI components other than REST.  The standard is pretty much REST and WebSockets for UI integration, with UI being a different layer (e.g., Angular). 

What do you mean by "just not EJBs"?  I use EJBs heavily.  Though, I'll strip out the remoting code if needed as this is dead code from 2002.  I only needed support to have it compile for now.  The only hard part about stripping it out is I don't have time to rewrite all its functionality using REST APIs.  But, the goal is REST/WebSockets only for client interfacing. 

For v4 we wanted to focus the programming model on CDI and not EJB, so right now there is no plans for adding EJB into v4. We appreciate that it might be a problem for applications that have evolved over a great deal of time, but we want to focus on programming models that fit with microservices in the cloud, at least from our perspective.

 

A quick google suggests the web profile includes "EJB 3.1 Lite", JPA and JTA, all of which I use.  Simply because of unit testing, I have been migrating away from not only remote interfaces, but explicit local interfaces, in favor of @LocalBean.  And I use JPA. 

https://jaxenter.com/introducing-the-java-ee-web-profile-103275.html

To be sure, I can see uses cases that don't require these, as I have another microservice that uses CrateDB instead of MariaDB.  For that, I don't use JPA, but just JDBC.  It seemed like a sacrifice in the beginning to not have JPA.  But, when you are using a high performance well indexed document database with sharding, you discover that data-modeling, and thus the benefits of JPA, are far less important.  This changes how you view transactions as well, since a commit, instead of encompassing multiple tables, will often only touch one table, which contains a document with a lot of depth.  

I have discovered that for now at least, a hybrid approach using traditional SQL (and thus JPA/JTA/EJBs) for ACID along with a document database gives you the best of both worlds.  I've seen a lot of requirements also including both. 

My apps all depend on the EJB Timer service excluded from the Web Profile.  I'm not sure why anyone would want to remove timers.  But missing this could be a show stopper for upgrading to V4.  Example use case: User registers.  That can be synchronously saved in DB.  But, propagating to another API requires scheduling, as other API may not alway be up.  I use a timer to handle in-memory task queuing.  In short, my apps do a lot asynchronously to ensure API calls are not blocking and communication availability with external APIs has fault-tolerance. 

Using timers in that way isn't something I've seen. Is it purely for delaying when an item of the queue is taken to be processed?

Is an external message queue, or something of that nature, an option? I'm curious in your case what happens if the service goes down with items on the in-memory task queue?

 

I have an app I haven't converted to microservice, yet, but plan to, that relies on MDBs that read a live in-memory queue and broadcast using Observables.  The queue is separately populated with live real-time data.  One impact is the UI is constantly updated via web sockets with live data that ultimately originates from an external streaming API, but transformed via the microservices app.  This requires queuing to handle the large quantity of real-time data in addition to the source of the data being properly loosely coupled from the web use case via the queues. 

Leaving out asynchronous or queuing is ignoring real-time applications and their requirements. These are not traditional monolithic use cases.  This is not tying into an enterprise bus.  It is leveraging modern web capabilities in a real-time world, connecting real-time APIs with web clients being a part of the channel in addition to automated clients (other consumers, service accouts), both for input and output, including the use of web sockets in addition to REST APIs. 

We are most definitely not leaving out asynchronous or queueing functionality from v4. In fact, they are critical pieces necessary for microservice development.

We already have support for our implementation of MDBs to provide message driven behavior, and we're looking at our options for integrating more asynchronous and reactive programming into v4.

[Erik]

On Tuesday, June 5, 2018 at 4:51:35 PM UTC-4, Ken Finnigan wrote:

On Tuesday, June 5, 2018 at 4:38:05 PM UTC-4, Erik wrote:

Ken, that is good news on the POM migration.  YAML changes sound like something we can easily work through.  TY for JDK 8 support.  The less day 1 change, the better. 

I could care less about JSF, JSP or any UI components other than REST.  The standard is pretty much REST and WebSockets for UI integration, with UI being a different layer (e.g., Angular). 

What do you mean by "just not EJBs"?  I use EJBs heavily.  Though, I'll strip out the remoting code if needed as this is dead code from 2002.  I only needed support to have it compile for now.  The only hard part about stripping it out is I don't have time to rewrite all its functionality using REST APIs.  But, the goal is REST/WebSockets only for client interfacing. 

For v4 we wanted to focus the programming model on CDI and not EJB, so right now there is no plans for adding EJB into v4. We appreciate that it might be a problem for applications that have evolved over a great deal of time, but we want to focus on programming models that fit with microservices in the cloud, at least from our perspective.

I guess I still need clarity because the Web Profile I referenced include "EJB 3.1 Lite".  Are you saying there won't even be stateless session beans or JPA entity beans?  

 

A quick google suggests the web profile includes "EJB 3.1 Lite", JPA and JTA, all of which I use.  Simply because of unit testing, I have been migrating away from not only remote interfaces, but explicit local interfaces, in favor of @LocalBean.  And I use JPA. 

https://jaxenter.com/introducing-the-java-ee-web-profile-103275.html

To be sure, I can see uses cases that don't require these, as I have another microservice that uses CrateDB instead of MariaDB.  For that, I don't use JPA, but just JDBC.  It seemed like a sacrifice in the beginning to not have JPA.  But, when you are using a high performance well indexed document database with sharding, you discover that data-modeling, and thus the benefits of JPA, are far less important.  This changes how you view transactions as well, since a commit, instead of encompassing multiple tables, will often only touch one table, which contains a document with a lot of depth.  

I have discovered that for now at least, a hybrid approach using traditional SQL (and thus JPA/JTA/EJBs) for ACID along with a document database gives you the best of both worlds.  I've seen a lot of requirements also including both. 

My apps all depend on the EJB Timer service excluded from the Web Profile.  I'm not sure why anyone would want to remove timers.  But missing this could be a show stopper for upgrading to V4.  Example use case: User registers.  That can be synchronously saved in DB.  But, propagating to another API requires scheduling, as other API may not alway be up.  I use a timer to handle in-memory task queuing.  In short, my apps do a lot asynchronously to ensure API calls are not blocking and communication availability with external APIs has fault-tolerance. 

Using timers in that way isn't something I've seen. Is it purely for delaying when an item of the queue is taken to be processed?

Is an external message queue, or something of that nature, an option? I'm curious in your case what happens if the service goes down with items on the in-memory task queue?

 

Basically, the same way SMTP handles priority queues to provide fault-tolerance.  Acts as a retry mechanism until it succeeds, and can lower priority of items after X retries to reduce frequency of retry.  This is how your emails sent will typically retry for 4 days before bouncing. If the server goes down, it can reconcile via the DB.  This would either be used for things that are already persisted and thus can be re-constructed from persisted data, or for things where the loss at server down if acceptable. 

If you like, at some point, I can blog on how I use it with sample code, because I created a highly re-usable simple pattern.  Once you have it, you can simply add tasks to it.  Initially, I put tasks I do not want to block the current HTTP thread.  Within it, there is a retry mechanism like SMTP.  The timer triggers the next retry.  Just like SMTP has intervals for handling its queues. 

I try to avoid persistent MQ queues as much as possible because they are heavy and cannot handle the large volumes of real-time data.  To be sure, though, the real time data is published to topic subscribed to by a persistence handler as well as real-time app handling. Each one can filter according to need.  I chose an architecture that uses in-memory MQ today, but has the option of adding remote listeners down the road if needed.  Not every real-time message is persisted.  Not every real-time message needs to be propagated to the app.  This comes down to "subscriptions" and filtering.  Not an MQ thing, and app content thing.  It's hard to explain in more detail without detailing the app.  But, it works really well, can handle high throughput very well, and with clustering via web sockets (Java to Java) is infinitely scalable to serve potentially an unlimited number of end-users or agents (ML/AI).  I can also blog on that pattern if you think it will help you or others. 

For the system I haven't converted to microservices yet, but plan to, that uses in-memory MQ and real-time data, here is some blogging on it:

http://www.eriksuniverse.com/developing-automated-trading-system.html

(Link button not working in FF, so may have to copy and paste)

In many ways it is primed for microservices conversion, only using a REST API and web sockets.  The UI, like my microservice apps, is Angular and separately deployed. 

[Ken Finnigan]

On Tuesday, June 5, 2018 at 9:35:20 PM UTC-4, Erik wrote:

On Tuesday, June 5, 2018 at 4:51:35 PM UTC-4, Ken Finnigan wrote:

On Tuesday, June 5, 2018 at 4:38:05 PM UTC-4, Erik wrote:

Ken, that is good news on the POM migration.  YAML changes sound like something we can easily work through.  TY for JDK 8 support.  The less day 1 change, the better. 

I could care less about JSF, JSP or any UI components other than REST.  The standard is pretty much REST and WebSockets for UI integration, with UI being a different layer (e.g., Angular). 

What do you mean by "just not EJBs"?  I use EJBs heavily.  Though, I'll strip out the remoting code if needed as this is dead code from 2002.  I only needed support to have it compile for now.  The only hard part about stripping it out is I don't have time to rewrite all its functionality using REST APIs.  But, the goal is REST/WebSockets only for client interfacing. 

For v4 we wanted to focus the programming model on CDI and not EJB, so right now there is no plans for adding EJB into v4. We appreciate that it might be a problem for applications that have evolved over a great deal of time, but we want to focus on programming models that fit with microservices in the cloud, at least from our perspective.

I guess I still need clarity because the Web Profile I referenced include "EJB 3.1 Lite".  Are you saying there won't even be stateless session beans or JPA entity beans?   

Whether EJB or EJB Lite, neither are currently planned for v4.

JPA with Hibernate is already part of v4

 

 

A quick google suggests the web profile includes "EJB 3.1 Lite", JPA and JTA, all of which I use.  Simply because of unit testing, I have been migrating away from not only remote interfaces, but explicit local interfaces, in favor of @LocalBean.  And I use JPA. 

https://jaxenter.com/introducing-the-java-ee-web-profile-103275.html

To be sure, I can see uses cases that don't require these, as I have another microservice that uses CrateDB instead of MariaDB.  For that, I don't use JPA, but just JDBC.  It seemed like a sacrifice in the beginning to not have JPA.  But, when you are using a high performance well indexed document database with sharding, you discover that data-modeling, and thus the benefits of JPA, are far less important.  This changes how you view transactions as well, since a commit, instead of encompassing multiple tables, will often only touch one table, which contains a document with a lot of depth.  

I have discovered that for now at least, a hybrid approach using traditional SQL (and thus JPA/JTA/EJBs) for ACID along with a document database gives you the best of both worlds.  I've seen a lot of requirements also including both. 

My apps all depend on the EJB Timer service excluded from the Web Profile.  I'm not sure why anyone would want to remove timers.  But missing this could be a show stopper for upgrading to V4.  Example use case: User registers.  That can be synchronously saved in DB.  But, propagating to another API requires scheduling, as other API may not alway be up.  I use a timer to handle in-memory task queuing.  In short, my apps do a lot asynchronously to ensure API calls are not blocking and communication availability with external APIs has fault-tolerance. 

Using timers in that way isn't something I've seen. Is it purely for delaying when an item of the queue is taken to be processed?

Is an external message queue, or something of that nature, an option? I'm curious in your case what happens if the service goes down with items on the in-memory task queue?

 

Basically, the same way SMTP handles priority queues to provide fault-tolerance.  Acts as a retry mechanism until it succeeds, and can lower priority of items after X retries to reduce frequency of retry.  This is how your emails sent will typically retry for 4 days before bouncing. If the server goes down, it can reconcile via the DB.  This would either be used for things that are already persisted and thus can be re-constructed from persisted data, or for things where the loss at server down if acceptable. 

If you like, at some point, I can blog on how I use it with sample code, because I created a highly re-usable simple pattern.  Once you have it, you can simply add tasks to it.  Initially, I put tasks I do not want to block the current HTTP thread.  Within it, there is a retry mechanism like SMTP.  The timer triggers the next retry.  Just like SMTP has intervals for handling its queues. 

I try to avoid persistent MQ queues as much as possible because they are heavy and cannot handle the large volumes of real-time data.  To be sure, though, the real time data is published to topic subscribed to by a persistence handler as well as real-time app handling. Each one can filter according to need.  I chose an architecture that uses in-memory MQ today, but has the option of adding remote listeners down the road if needed.  Not every real-time message is persisted.  Not every real-time message needs to be propagated to the app.  This comes down to "subscriptions" and filtering.  Not an MQ thing, and app content thing.  It's hard to explain in more detail without detailing the app.  But, it works really well, can handle high throughput very well, and with clustering via web sockets (Java to Java) is infinitely scalable to serve potentially an unlimited number of end-users or agents (ML/AI).  I can also blog on that pattern if you think it will help you or others. 

For the system I haven't converted to microservices yet, but plan to, that uses in-memory MQ and real-time data, here is some blogging on it:

http://www.eriksuniverse.com/developing-automated-trading-system.html

(Link button not working in FF, so may have to copy and paste)

In many ways it is primed for microservices conversion, only using a REST API and web sockets.  The UI, like my microservice apps, is Angular and separately deployed. 

Thanks for the detailed explanation Erik, and I do think others in the community would find it interesting to hear about what you've done here.

In terms of what's provided in v4, would it be possible for you to include the Quartz library into your application to provide the timing functionality you need?

In v4 we're wanting to have less offered with the project, but make it easier to bring your own library for pieces of functionality we don't include.

[dansiviter]

I think it's pretty evident from this we need more clarity on the plans. Not having EJB's (especially the Timer Service/@Schedule and Message Driven Beans) in v4 will be a pain which will require a fair amount of re-engineering so the sooner this is clarified the sooner we can address and be ready.

For v2, will that be kept in line with Wildfly? Especially as WF13 is now available.

Thanks

[Ken Finnigan]

We have an implementation of MDBs available within v4, as we see that as appropriate. However, we haven't incorporated the remainder of EJBs so we can focus on the CDI programming model. Some items, such as timers, can be replaced with libraries, in the case of timers it's Quartz.

Can you elaborate on what you'd like to see to assist with clarity? I stated above that v4 intends to incorporate, and does already, everything up to the Web Profile but excluding JSF, and EJBs, apart from MDBs which are included.

Right now we don't have definitive plans over when/if v2 would update to Java EE 8 versions of dependencies by updating the underlying WildFly it's based on.

Ken

[Erik]

Ken, I'd be willing to consider Quarts if that was all that was needed to migrate to V4, presuming that plays well with injection and other requirements.  

I think we still need more clarity on EJBs.  You said that V4 includes JPA with Hibernate.  Isn't that EJBs?  Entity beans are a part of EJB.  So that only leaves the question of Stateless session beans, which I use a lot, along with their transaction demarcation.  E.g., 

@TransactionAttribute(REQUIRES_NEW)

https://docs.oracle.com/javaee/6/tutorial/doc/bncij.html

I can't remember the details, but I think there are cases where I use EJB instead of CDI, where I could only get EJB to give the behavior I needed, such as @ApplicationScoped vs @Singleton.

It sounds like you're saying V4 would support @Singleton, but not @ApplicationScoped. 

And what about @LocalBean and  @EJB injection? 

What would help is a list of Java packages and annotation that will not be included.  This would allow us to assess more precisely how this impacts our code and if V4 is a direction that makes sense for us.  Whether we choose to adapt to V4 or choose another option, we'll need to plan this a bit early to manage our current investment and direction. 

[Ken Finnigan]

On Wednesday, June 6, 2018 at 1:32:47 PM UTC-4, Erik wrote:

Ken, I'd be willing to consider Quarts if that was all that was needed to migrate to V4, presuming that plays well with injection and other requirements.  

I think we still need more clarity on EJBs.  You said that V4 includes JPA with Hibernate.  Isn't that EJBs?  Entity beans are a part of EJB.  So that only leaves the question of Stateless session beans, which I use a lot, along with their transaction demarcation.  E.g., 

@TransactionAttribute(REQUIRES_NEW)

https://docs.oracle.com/javaee/6/tutorial/doc/bncij.html

Entity Beans are part of JPA and not EJB specs. v4 providing JPA means you can use Entity Beans.

@TransactionAttribute comes from the EJB spec, so yes that wouldn't be there. However, the JTA spec provides the @Transactional annotation that works well with CDI Beans.

If @Transactional doesn't quite fit the bill of what you need, we can always look at creating an equivalent to @TransactionAttribute that's implemented with CDI and JTA underneath.

 

I can't remember the details, but I think there are cases where I use EJB instead of CDI, where I could only get EJB to give the behavior I needed, such as @ApplicationScoped vs @Singleton. 

It sounds like you're saying V4 would support @Singleton, but not @ApplicationScoped. 

@ApplicationScoped comes from CDI, so yes that would be present.

 

And what about @LocalBean and  @EJB injection? 

If you mean the ability to inject EJB beans into other classes. Then you're correct in saying that wouldn't be present. However, CDI Beans can be @Inject'd as a replacement usage.

 

What would help is a list of Java packages and annotation that will not be included.  This would allow us to assess more precisely how this impacts our code and if V4 is a direction that makes sense for us.  Whether we choose to adapt to V4 or choose another option, we'll need to plan this a bit early to manage our current investment and direction. 

Will take that idea back to the team and see what we can do.

[Ladislav Thon]

Terminology clarification: there are no entity beans! The "entity bean" term comes from EJB pre-3.0 era. WildFly removed entity beans entirely (both CMP and BMP) in version 10 (together with a couple of other things). I'm pretty sure that noone who uses WildFly Swarm / Thorntail actually uses EJB entity beans; everyone uses JPA. JPA continues to be supported in Thorntail v4.

LT

--

You received this message because you are subscribed to the Google Groups "Thorntail" group.
To unsubscribe from this group and stop receiving emails from it, send an email to thorntail+...@googlegroups.com.
To post to this group, send email to thor...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/thorntail/0eef5c2b-84e0-4edc-93f5-8f8ef1a05e73%40googlegroups.com.

[Erik]

No reference to the spec?  Beginning with the definition of "bean" in Java, and understanding that all Entity classes are beans, whether JPA @Entity beans, or CMP or BMP.  So, yes, there are and will always be entity beans.  And, yes, you are correct that WildFly does not support EJB 2.1 Entity Beans, or specifically, CMP or BMP. 

Obviously, if using JPA, you are not using the deprecated BMP and CMP.  But, you are still using beans "with a proxy", which is conceptually the same as "container managed".  In Hibernate terms, attached (where a proxy is provided) versus detached. 

It is pointless to split hairs.  The EJB 3.1 spec includes Entity Beans.   While not specifically addressing JPA as that has its own specifications, it does clarify when referring to EJB 2.1 entities, to avoid confusion with JPA specifications (e.g., JSR 338). 

I was on a team that was an early adopter of EJB 3 in 2006, migrating from 2.1, and have used JPA since.  Never until now have I heard people try to claim that beans annotated @Entity are not "entity beans".  We all recognized JPA as a part of the new EJB 3 release, and a replacement for the CMP/BMP deprecated.  Its integration with Session beans (CMT), was functionally equivalent, including support for declarative transactions on methods (https://docs.oracle.com/javaee/6/tutorial/doc/bncij.html.)  I have always used JPA since 2006, yet considered it EJB 3 due to its replacement for EJB 2.1 on the entity side, continued integration with Session beans and declarative transactions.  Even Oracle, in this URL to JPQL, seems to agree that JPA is included in EJB 3:

https://docs.oracle.com/html/E13946_01/ejb3_langref.html
 
Rather than splitting hairs on definitions, how about using specific terminology to define exclusions, such as a list of annotations, classes or packages?  Much better to be specific with exclusions and avoid the terminology debate entirely. 

To be sure, once Ken clarified that Session beans would not be included, but there were intended functional equivalents for transaction demarcation using CDI, I understood the impact better.  In the end, functionally, it sounds like the only features in "EJB" I'm still using today are session beans for declarative transaction demarcation on methods with @EJB injection, timers and MDBs.  It sounds like CDI has progressed to try to replace these things with counterparts.  

I just have not had great success swapping EJB with CDI equivalents so far, with unexpected consequences when I tried, causing me to continue to use EJB annotations instead.  Perhaps it is easier if creating an app from scratch, than migrating an existing one where the two need to play together for awhile without breaking the app. 

[dansiviter]

Sure, I mean more of a breakdown of the JEE spec that will or won't be in v4 and available somewhere centrally (thortail.io?) rather than digging into Groups replies. You're committed to Microprofile, and v2 was committed to JEE7 but the water is now becoming more muddy.

Also, we could use Quartz, but the reason we didn't use it is because it's frankly an overkill and overly complex. EJB Timer Service fit the bill nicely.

[Ken Finnigan]

Dan,

We're definitely intending to update v4 to utilize Java EE 8/Jakarta EE pieces, there's already some issues around that created.

We can certainly look to create a more delineated definition of what is included/proposed and what isn't.

If Quartz is too complex, and you need something equivalent to EJB Timer Service it would be great if you could create an GH Issue outlining what you'd like to see. Maybe we can create a lightweight timer atop CDI that fits most needs.

Ken

[Ladislav Thon]

OK, I agree my reply wasn't particularly useful, especially for people who are well-versed in Java EE. I agree that clearly JPA @Entity classes are beans per the JavaBeans specification, but also clearly they are not EJB Entity Beans per the EJB specification. The EJB and JPA specifications, as far as I know, intentionally use the word "entity" when refering to JPA entities, instead of "entity bean", and they also use "EJB 2.1 entity bean".

I personally also tend to refer to JPA entities as just entities, leaving the term "entity beans" for EJB 2.1 entity beans. We might disagree on this terminology usage, but I hope we'd agree that it's a good distinction to make in this discussion.

To try to be a little more helpful, here's my understanding about the Thorntail v4 scope. It's web profile with these exceptions:

• JSF: not included
• EJB:
• message-driven beans: included, but not entirely happy about having to bring in the entire javax.ejb package because of this and maybe will develop something new
  
• entity beans: should be clear per above, but I'll repeat: EJB 2.1 entity beans not included, JPA included
  
• session beans: the point of contention; current plan is to not include them, but I agree it would be nice to have some kind of migration story

LT

To view this discussion on the web visit https://groups.google.com/d/msgid/thorntail/e946a0cc-52a0-4eb9-854e-4ee72fbc3b6e%40googlegroups.com.

[Erik]

BTW, one lesson I learned is that, at least with CrateDB, I don't need JPA.  Sharing this because I see you plan to support Hibernate OGM to allow JPA for no SQL data stores.  And I'd like to be helpful to the project if I can help to reduce your priorities. 

Referencing 14.2 at https://docs.thorntail.io/4.0.0-SNAPSHOT/#component-jaxrs

I'd like to share how CrateDB worked out for me, because if others make the same discovery, perhaps you won't need to spend time on OGM in V4, freeing up time for the more critical things, like smoothing V2 migration.

CrateDB supports JDBC (SQL with very robust JSON document support), but not JPA.  You can create tables, and you can store documents in tables.  Documents can have pre-defined structure, and dynamic (created at runtime) structure.  The SQL works with documents, including querying dynamically created fields in your WHERE clause, though you can only index/shared pre-defined fields.  Your horizontally clustered indexes can be on document fields.  This leads to HIGHLY AGILE development where you are no longer spending much time on data modeling, and instead, creating new capabilities rapidly on the fly in code.  It is the ultimate in extensibility and performance combined.  The only thing this world lacks is ACID, or specifically, immediate consistency for non-PK queries and atomic transactions.  To obtain infinite horizontal scalability, you need eventual consistency on non-PK queries.  Thus, the continued need for JPA with databases such as MariaDB (Galera for clustering), for use cases such as traditional OLTP.  Atomic transactions are less important with document databases because things that used to span many tables are all in one document, thus one commit to one row. 

https://www.eriksuniverse.com/test-galera-kubernetes.html

The end result is you do not need JPA for this use case, as it would just get in the way.  Instead, you still have your POJO object graph, you use something like Gson to marshal to/from JSON, which is itself very dynamic.  Thus, in Java, your document can continue to be represented by a set of related classes -- an object graph. 

Despite some lackings and gotchas you have to work around, in practice, Gson is the best tool I know of for this new world of rapidly dynamically changing, yet heavily indexed and sharded and horizontally scaled document database ideal for microservices.  It cannot replace JAXB for certain uses cases, such as marshalling XML, which may be a requirement when receiving documents from other APIs.  But, in a dynamic document world, Gson rules. 

Let me give you an example of a use case, so you can see why JPA would not be needed:

You have the need to create a new thing to persist.  Let's call it a "Thorntail feature request".  I can simply define a new document type, which in DB terms is just a new field value in an already indexed and sharded field.  Let's call that value "TTFR".  Your TTFR document can have new fields other document types don't have, which you would put in your dynamic extensions fields (fields created in the table for the purpose of dynamically creating new sub-documents), though it can also re-use common pre-existing fields (createdBy might pre-exist and thus be indexed for fast querying).  In Java, these new fields can be represented by as many POJOs as you want; and, can include collections.  Because of the indexing and sharing on the document type, this has the performance effect of creating a new table and indexes in the DB dedicated to just TTFR docs.  But, you don't need to change the data model at all!  In fact, if you expose your document extensions capability via a REST API, you can extend it completely externally, without ever touching your code in your microservice! 

I'd be pulling my hair out trying to map this to JPA if I tried, with little to no benefit.  And, that certainly wouldn't help the external consuming client apps using REST to CRUD the documents. 

You'll obviously have to get input from other "NoSQL" users (which should probably be called "non-ACID document DBs" as CrateDB proved you can continue to use SQL).   

To be sure, I continue to depend on JPA for traditional SQL for ACID use cases, and will continue that dependency for the foreseeable future.  I just keep ACID/JPA separate from highly scalable document store use cases, in so much that they are not in the same microservice deployments.  To the extend they interact, they do that through REST APIs. 

I am so insanely happy with CrateDB and this overall model, I can never imagine ever using JPA with a document or key store.

[dansiviter]

I've just given v4.0.0-SNAPSHOT a go and it's certainly start fast; just a few seconds! Unfortunately, I couldn't get JAX-RS Application class to be discovered, but as it's only early days I'm not overly concerned.

As for EJB timer service, it seems Apache DeltaSpike have a similar CDI based functionality (which does use Quartz under the hood) so that might help ease transition somewhat.

[Ken Finnigan]

JAX-RS Application classes should definitely be discovered. I only wonder if maybe your project has that in a dependent jar and not the project itself?

At any rate, it's something that should definitely work right now.

Ken

[Bob McWhirter]

The artifact with the JAX-RS classes needs a beans.xml to be recognized as a CDI archive (even if you don’t do any other CDI stuff).

-Bob

--
You received this message because you are subscribed to the Google Groups "Thorntail" group.
To unsubscribe from this group and stop receiving emails from it, send an email to thorntail+...@googlegroups.com.
To post to this group, send email to thor...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/thorntail/6b1742f7-5e68-45c4-a9b3-bf52ce48660e%40googlegroups.com.

[dansiviter]

Kudos: the beans.xml fixed it.