maven-based quarkus classloader

[William Burke]

I was thinking that new packaging possibilities might open up if a Quarkus could use maven artifact identifiers to define its classpath rather than bundling all dependencies in a legacy, fat, or quarkus app directory.  Much like JBoss Modules could reference maven artifact identifiers.

So, a Quarkus app jar would only have the .class files of the project and contain all dependencies within a manifest/dependencies.txt within the .jar.  Dependency jars would be resolved by looking in $HOME/.m2/repository by default, or via a specified property or env var.  i.e.

$ java -jar myapp.jar -Dmaven.root=/var/maven/repository

$ java -jar myapp.jar -Dmaven.root=http://repo.org

--

Bill Burke

Red Hat

[David Lloyd]

I like it (naturally ;)). One thing worth noting is that, like JBoss Modules, the resolution would be considerably less robust than what the actual Maven Resolver does. Not just because we wouldn't care about transitive dependencies, but also because we'd similarly want to hand-roll processing of settings and downloading (to minimize external dependencies from the foundational class loading).

Also we have to consider the case where we transform dependencies, in which case we'd have to bundle the transformed artifact regardless. I don't know how frequently we actually do this though, if we do it at all.

--

- DML • he/him

[Georgios Andrianakis]

My question is really say we had that today, what would we use it for?

--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CANghgrSitCz6LhmWLQMXqLc6GJ56MLUAvXZgGLa1w%2BxEP%2BsRAg%40mail.gmail.com.

[David Lloyd]

That's a reasonable question... off the cuff I'd say, smaller packaging (i.e. disk space), maybe better sharing/layering opportunities for container images, maybe faster classloading (thus startup) if we go the full JBoss Modules style package index route (compared to flat classpaths).  We could also utilize a more sophisticated front-end class loading mechanism to dynamically JPMS-ize some or all dependencies in the future, making it easier to add an access-control wedge for reflection, FFM, etc.

It seems to me that a few different, disparate issues are converging on "let's have better class loading", so I'm thinking along the lines of "what can we do more easily if we took an architectural approach to class loading?", and this is certainly something that should be easy to do in such a scenario. However, this of course only covers one (or a subset of) possible packaging option(s).

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CALeTM-nkeUMXFO9B5WpTtA7s6YxJTiT-PndBRw2AzFu0Aq%2BeGQ%40mail.gmail.com.

--

- DML • he/him

[Max Rydahl Andersen]

Jbang basically allows this. 

But that's external launch. 

You are suggesting the bootloader of quarkus natively understands it. 

Could work - what do you expect to happen if you just have the jar ? That the bootloader starts fetching dependencies or fail fast ?

/max

https://about.me/maxandersen

--

You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAL%3DE%3DjSuuKN%3DrfA-a4OjT_bfGdjxKnLU2KRHpEi0fbvda7acFA%40mail.gmail.com.

[Max Rydahl Andersen]

Transform dependencies would be a challenge though. 

As Georgios ask - what would this extra level of indirection bring? Shared maven artifacts when running different apps on same machine/container ?

/max

https://about.me/maxandersen

--

You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CANghgrSitCz6LhmWLQMXqLc6GJ56MLUAvXZgGLa1w%2BxEP%2BsRAg%40mail.gmail.com.

[David Lloyd]

We could do whatever we want. Bill's second example sort of implies fetching things. I think that might be nifty (JBoss Modules basically has this capability and it seems to be OK).

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAHL%3D7VsSVS%3DrQXBdRZSFrpvE9%3DFJuO_3kqfCjXdwut6q_vtcTA%40mail.gmail.com.

--

- DML • he/him

[Georgios Andrianakis]

How do we feel that this would fit in a container environment?

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CANghgrR1OguhZ-i4ECXr1x_V7RASuYe1yB6RRgh%2B83OJgBh1fg%40mail.gmail.com.

[Marco Bungart]

This is what I throught as well. The concept of fetching artifacts at startup seems very counter-intuitive to the immutability of containers. Like we do not install software in a running container (we would do this at image buildtime), I would not want to fetch artifacts at startup. Then there is of course the challenge with air-gapped environments. And lastly, I do not like the idea that the artifacts might change depending on when I started the application (especially during development, when we use snapshot artifacts that can be overridden, this sounds like a lengthy debugging session).

Cheers,

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CALeTM-kFJN6U6_bmsVt_HpJfkzKeOUAXYguA8qaK1svzw%2BH-jA%40mail.gmail.com.

[David Lloyd]

Referencing the existing implementation that WildFly is using, none of these issues necessarily exist. Fetching at run time would be optional and situational (see Bill's example).

For snapshots, would this be different/worse than any other project that is built and run?

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/4906f145-5ed8-46ef-903d-defe3168dd8d%40googlemail.com.

--

- DML • he/him

[Alexey Loubyansky]

What I’m missing is how using Maven artifact IDs instead of fast-jar layout could benefit classloading architecture.

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CANghgrTWLpJBaLBSBmdCcwN9ePweJnS37eNiDvO7to7i_CUQFg%40mail.gmail.com.

[David Lloyd]

I'm thinking of it the other way around: that certain improvements to the class loading architecture could more easily enable using Maven artifact ID-based packaging as an alternative, as well as being able to benefit existing fast-jar or uber-jar layouts. In my mind, revisiting class loading architecture is inevitable (thinking of JPMS and other things), so I'm looking at it through that lens.

--

- DML • he/him

[Marco Bungart]

Re snapshots: Yes. When the image is built, it is built, with exactly one version of each dependency. The dependencies do not change just because I start the image at a different point in time.

If we do not fetch at image startup, I do not see how the package size might get smaller. Sure - we could have a "common layer" with an abundance of dependencies and then always layer our applications over that. This sounds, however, as if we would increase the overall image size. Plus, we suddenly have libraries on the classpath that we "do not use", but might creep into our application anyway (e.g. through service loading).

[George Gastaldi]

Would that impact Quarkus applications using Gradle too? 

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CANghgrTMaaF6JXm-4F3qa%2BhxJihS%3DddenHzSS7vY_RNui0j5Ug%40mail.gmail.com.

[David Lloyd]

I would imagine so. You'd still need to consider Maven settings to locate the local and/or remote repository (and optional proxy settings, etc.). But the build tool otherwise shouldn't matter much I would imagine.

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAMmsnJjMhNzxA_bH_qotMGwGfFPJ8r7Y0aArUZfpSVsnJBxeWg%40mail.gmail.com.

--

- DML • he/him

[David Lloyd]

A common layer is what I was thinking of. I believe we have a number of common library dependencies that will always be needed/included. Of course this is all off-the-cuff spitballing as a reaction to Bill's idea (maybe he has a comment?) so I have no idea whether it would actually be worthwhile. But it seems to me that it *could* be.

--

- DML • he/him

[Alexey Loubyansky]

I would separate Maven artifact resolution from classloading architecture enhancements. I think the two shouldn’t be related at all.

I mean dependency information is important in hierarchical class loading architectures but this is not something this specific layout enables compared to the already existing ones. So I’d separate these two discussions.

[Max Rydahl Andersen]

> Would that impact Quarkus applications using Gradle too?

I would assume it would be separate-complementary.

i.e. jbang runs maven, gradle, etc. just fine.

This here would just be packaging of dependencies.txt and then a resolution against
maven repo (local and/or remote)

And it would be pure runtime scoped deps with no transitive deps.
-max

>>>>>> <https://groups.google.com/d/msgid/quarkus-dev/CANghgrSitCz6LhmWLQMXqLc6GJ56MLUAvXZgGLa1w%2BxEP%2BsRAg%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>>>> .

>>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "Quarkus Development mailing list" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to quarkus-dev...@googlegroups.com.
>>>>> To view this discussion on the web visit

>>>>> https://groups.google.com/d/msgid/quarkus-dev/CALeTM-nkeUMXFO9B5WpTtA7s6YxJTiT-PndBRw2AzFu0Aq%2BeGQ%40mail.gmail.com
>>>>> <https://groups.google.com/d/msgid/quarkus-dev/CALeTM-nkeUMXFO9B5WpTtA7s6YxJTiT-PndBRw2AzFu0Aq%2BeGQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>>> .

>>>>>
>>>>
>>>>
>>>> --
>>>> - DML • he/him
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Quarkus Development mailing list" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to quarkus-dev...@googlegroups.com.
>>>> To view this discussion on the web visit

>>>> https://groups.google.com/d/msgid/quarkus-dev/CANghgrTWLpJBaLBSBmdCcwN9ePweJnS37eNiDvO7to7i_CUQFg%40mail.gmail.com
>>>> <https://groups.google.com/d/msgid/quarkus-dev/CANghgrTWLpJBaLBSBmdCcwN9ePweJnS37eNiDvO7to7i_CUQFg%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>> .

>>>>
>>>
>>
>> --
>> - DML • he/him
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Quarkus Development mailing list" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to quarkus-dev...@googlegroups.com.
>> To view this discussion on the web visit

>> https://groups.google.com/d/msgid/quarkus-dev/CANghgrTMaaF6JXm-4F3qa%2BhxJihS%3DddenHzSS7vY_RNui0j5Ug%40mail.gmail.com
>> <https://groups.google.com/d/msgid/quarkus-dev/CANghgrTMaaF6JXm-4F3qa%2BhxJihS%3DddenHzSS7vY_RNui0j5Ug%40mail.gmail.com?utm_medium=email&utm_source=footer>
>> .

>>
>
> --
> You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAMmsnJjMhNzxA_bH_qotMGwGfFPJ8r7Y0aArUZfpSVsnJBxeWg%40mail.gmail.com.

[Alexey Loubyansky]

On Wed, May 22, 2024 at 10:07 PM Max Rydahl Andersen <mand...@redhat.com> wrote:

> Would that impact Quarkus applications using Gradle too?

I would assume it would be separate-complementary.

i.e. jbang runs maven, gradle, etc. just fine.

This here would just be packaging of dependencies.txt and then a resolution against
maven repo (local and/or remote) 

And it would be pure runtime scoped deps with no transitive deps.

That's where it probably diverges from the other idea of classloading redesign to move away from the flat classloading model.

This idea is about flattening a dependency graph to a list. dependencies.txt is pretty much as flat as it can get.

Non-flat classloading architectures need to take artifact dependencies into account, so that info would have to come from somewhere. Re-resolving a dependency graph from POMs would kind of undermine the idea of a flat artifact list.

We actually have a better alternative to dependencies.txt - serialized ApplicationModel, which contains comprehensive dependency info but not in clear text. But anyway, we shouldn't mix two topics in one thread, imo.

I am still not clear what the actual benefits of Bill's proposal would be besides loading JARs from a shared location - local Maven repo. Is this what this is actually about?

It wouldn't be that straightforward though, an application would have to keep bytecode enhanced artifacts in a location that would have a higher priority over the local Maven repo.

[William Burke]

On Wed, May 22, 2024 at 4:38 PM Alexey Loubyansky <alexey.l...@redhat.com> wrote:

I am still not clear what the actual benefits of Bill's proposal would be besides loading JARs from a shared location - local Maven repo. Is this what this is actually about?

It wouldn't be that straightforward though, an application would have to keep bytecode enhanced artifacts in a location that would have a higher priority over the local Maven repo.

Benefits?  

* Same benefits as a shared library. 

* Smaller disk footprint, think of all our integration tests in quarkus git repo that copy all the jars needed for each integration test.

* Shared container mounts

* Repos store other things than the .jar that might be useful

* Very thin distributions

IMO an application becomes much more structured.

 

[Stuart Douglas]

On Thu, 23 May 2024 at 02:42, David Lloyd <david...@redhat.com> wrote:

That's a reasonable question... off the cuff I'd say, smaller packaging (i.e. disk space), maybe better sharing/layering opportunities for container images, maybe faster classloading (thus startup) if we go the full JBoss Modules style package index route (compared to flat classpaths).  We could also utilize a more sophisticated front-end class loading mechanism to dynamically JPMS-ize some or all dependencies in the future, making it easier to add an access-control wedge for reflection, FFM, etc.

Fast-Jar already has package indexing that mostly works the same as in jboss-modules. It is written into the serialized data so the full index is known at build time.

 

It seems to me that a few different, disparate issues are converging on "let's have better class loading", so I'm thinking along the lines of "what can we do more easily if we took an architectural approach to class loading?", and this is certainly something that should be easy to do in such a scenario. However, this of course only covers one (or a subset of) possible packaging option(s).

What exactly are you proposing around class loading? Has something changed with native mode that would allow support for non-flat class paths?

Stuart

 

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CANghgrTWLpJBaLBSBmdCcwN9ePweJnS37eNiDvO7to7i_CUQFg%40mail.gmail.com.

[Stuart Douglas]

On Thu, 23 May 2024 at 11:06, William Burke <bbu...@redhat.com> wrote:

On Wed, May 22, 2024 at 4:38 PM Alexey Loubyansky <alexey.l...@redhat.com> wrote:

I am still not clear what the actual benefits of Bill's proposal would be besides loading JARs from a shared location - local Maven repo. Is this what this is actually about?

It wouldn't be that straightforward though, an application would have to keep bytecode enhanced artifacts in a location that would have a higher priority over the local Maven repo.

Benefits?  

* Same benefits as a shared library. 

So basically if you have lots of quarkus applications deployed in a non-container environment it would save a little bit of disk space?

 

* Smaller disk footprint, think of all our integration tests in quarkus git repo that copy all the jars needed for each integration test.

That only really applies to us building quarkus though, I don't know if application developers really care about this when you are just maintaining a single application.

 

* Shared container mounts

* Repos store other things than the .jar that might be useful

* Very thin distributions

IMO an application becomes much more structured.

To be honest I don't really see this as being super useful, especially not when we are talking about cloud deployment.

For EAP this was a big improvement, because EAP was distributed as a 100mb+ zip file, so being able to re-use jars that were already in your maven repo was a big saving. Quarkus already uses the jars in your maven repo, it just assembles them into the app, so I don't know if the benefits are there, compared to the extra complexity.

Also this will mean that your apps will need to potentially load maven resolver classes at runtime, so it will likely also hit startup speed and memory usage.

Stuart

 

 

--

Bill Burke

Red Hat

--

You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAL%3DE%3DjSLkJjpxAm_B8eb5Pdecx6BwcjD-7UT4OS%2BmNNWS8VWpA%40mail.gmail.com.

[David Lloyd]

On Wed, May 22, 2024 at 10:19 PM Stuart Douglas <sdou...@redhat.com> wrote:

On Thu, 23 May 2024 at 02:42, David Lloyd <david...@redhat.com> wrote:

That's a reasonable question... off the cuff I'd say, smaller packaging (i.e. disk space), maybe better sharing/layering opportunities for container images, maybe faster classloading (thus startup) if we go the full JBoss Modules style package index route (compared to flat classpaths).  We could also utilize a more sophisticated front-end class loading mechanism to dynamically JPMS-ize some or all dependencies in the future, making it easier to add an access-control wedge for reflection, FFM, etc.

Fast-Jar already has package indexing that mostly works the same as in jboss-modules. It is written into the serialized data so the full index is known at build time.

Great!

 

It seems to me that a few different, disparate issues are converging on "let's have better class loading", so I'm thinking along the lines of "what can we do more easily if we took an architectural approach to class loading?", and this is certainly something that should be easy to do in such a scenario. However, this of course only covers one (or a subset of) possible packaging option(s).

What exactly are you proposing around class loading?

I'm not proposing anything, yet.

 

Has something changed with native mode that would allow support for non-flat class paths?

Native compilation with qbicc has always supported non-flat class paths, but that's likely to remain experimental for the foreseeable future. For GraalVM, I don't recall; I thought it was discussed at some point.

--

- DML • he/him