Quarkus 2.x: Decomposed platform model proposal

[Alexey Loubyansky]

I’ve been researching an alternative to the current Quarkus platform model lately. And here is my proposal for Quarkus 2.x.
Just in case, the concept of the platform and how it is different from the rest of the ecosystem is described in [1]

1. MOTIVATION

The current platform is released as a single BOM (quarkus-universe-bom). While it seems simple to have a single BOM import covering all the platform extensions in a Quarkus app, there are pretty serious issues with this approach.

1.2. PLATFORM AND NON-PLATFORM DEPENDENCIES

The platform BOM is pretty big, e.g. [2]. And it keeps growing. If an app is using a few (or even a bit more than a few) platform extensions then the platform dependency constraints that are not relevant to those extensions (which practically always be a huge amount) will still be enforced on the app. That could potentially create conflicts for other (non-Quarkus) libraries or even Quarkiverse extensions that might not be trivial to debug and resolve.

E.g. if an app isn’t using Camel extensions (Camel’s scope is particularly large), all the Camel dependency constraints will still be enforced on the app. If there is a library or a Quarkiverse extension that is not compatible with Camel’s constraints, it might not be easy to make it work in the app. It’s not impossible. The necessary dependency constraints could still be re-written in the app config. But it might be a painful task and surely not a great UX.

With the growing platform and even quicker growing Quarkiverse we will inevitably see more and more incompatibility issues of that nature between the platform and non-platform extensions. The only recommendation we could give to users in those cases would be to override the platform constraints in their apps. Which sounds neither right nor safe.

1.3. PLATFORM RELEASE

The all-inclusive platform BOM implies a monolithique platform. I.e. the set of members is pretty much fixed and everyone has to be ready for the platform release. If any one member isn’t ready (e.g. not passing the tests, etc) this blocks the release of the whole platform. Excluding that member from one platform release and re-including it in the next one isn’t really acceptable in the current platform model. Simply because upgrading to the next platform version may break applications.

2. PROPOSAL

Simply speaking, the idea is to decompose the all-inclusive platform BOM into member (or tech) -specific BOMs (e.g. core BOM, Camel BOM, Kogito BOM, etc) and let applications import the relevant BOMs instead of everything at once.
The dependency constraints across all the member-specific BOMs must still be aligned, i.e. there shouldn’t be conflicts between the member BOMs to make sure that including, e.g., Camel and Kogito in the same app will still work.

To keep the member BOMs aligned we would still have to use the platform BOM generator [3] (as we do today). But after the all-inclusive set of platform dependency constraints have been generated, instead of persisting and releasing it as a single BOM (quarkus-universe-bom), this generated set of constraints will be split back into member-specific BOMs and those member BOMs will released (assuming all the platform integration tests have passed) instead of the quarkus-universe-bom.

That way, instead of a monolithic platform, we would have a stack of aligned platform member BOMs. That could be imported in apps in any order w/o creating conflicts, given that they belong to the same globally aligned set of dependency constraints.

2.1. EFFECTS ON THE DEV TOOLS AND APP CONFIG

If an app needs to add a platform extension that’s not in the quarkus-bom, the corresponding platform BOM will need to be imported in addition to the quarkus-bom. The dev tools will be able to suggest and/or actually import the right version of the right BOM. Users who prefer writing POMs/gradle scripts manually will have to consult the dev tools suggestions or platform docs for the right BOM version to import. Which is definitely more difficult than having a single import but, otoh, is pretty much a usual development practice.

2.2. PLATFORM VERSION UPDATES

If an app is importing more than one platform BOM, updating to the next platform release *may* imply updating more than one BOM version. The dev tools will be able to check for and list the available updates, and also apply them, if requested.
Users who prefer editing project configs manually would be able to use the tools to list the available updates and apply them manually.

Manual project config maintenance becomes more error prone with this model. The good news is that, at app build time, we would be able to check whether the imported platform BOMs belong to the same platform stack release and are a tested combination. If it’s not the case we would display a clear error/warning message indicating which BOM versions were found to be misaligned and how the versions should be adjusted to have a properly aligned stack.

3. APPLICATION POM EXAMPLES

Applications using only core extensions (found in the quarkus-bom) instead of importing

<dependency>
  <groupId>io.quarkus</groupId>
  <artifactId>quarkus-universe-bom</artifactId>
  <version>1.13.3.Final</version
  <type>pom</type>
  <scope>import</scope>
</dependency

like today would be importing the quarkus-bom, e.g.

<dependency>
  <groupId>io.quarkus.platform</groupId>
  <artifactId>quarkus-bom</artifactId>
  <version>1.13.3.Final</version
  <type>pom</type>
  <scope>import</scope>
</dependency

Note: the groupId if io.quarkus.platform. It doesn’t have to be that way, but either that or the version has to be different from the original core release because we’ll need to attach some platform stack metadata as part of the platform release to each member of the stack.

Applications that use, e.g. OptaPlanner extensions (currently included into the quarkus-universe-bom) instead of importing

<dependency>
  <groupId>io.quarkus</groupId>
  <artifactId>quarkus-universe-bom</artifactId>
  <version>1.13.3.Final</version
  <type>pom</type>
  <scope>import</scope>
</dependency

will be importing the quarkus-bom and the optaplanner-bom (produced by the platform bom generator), e.g.

<dependency>
  <groupId>io.quarkus.platform</groupId>
  <artifactId>quarkus-bom</artifactId>
  <version>1.13.3.Final</version
  <type>pom</type>
  <scope>import</scope>
</dependency
<dependency>
  <groupId>io.quarkus.platform</groupId>
  <artifactId>optaplanner-bom</artifactId>
  <version>8.5.0.Final</version
  <type>pom</type>
  <scope>import</scope>
</dependency>

4. PoC

I do have a basic PoC of this model and even tried to record a video this morning but for some reason ran into issues with screen sharing and switching between windows. I could look into that later, if necessary.

Otherwise, let me know what you think. Especially, if it doesn't seem like a good idea. Thanks!

[1] https://quarkus.io/guides/platform
[2] https://repo1.maven.org/maven2/io/quarkus/quarkus-universe-bom/1.13.3.Final/quarkus-universe-bom-1.13.3.Final.pom
[3] https://github.com/quarkusio/quarkus-platform-bom-generator

[Sanne Grinovero]

Hi Alexey,

big +1 from me to challenge the status quo, the huge BOM is very problematic and I love you're exploring this.

I do think you're on the right track conceptually, but I wonder if we could completely hide the notion of the "multiple fine-grained BOMs" you're proposing?

Ideally each extension should have an implicit BOM, which is automatically applied when the extension is imported.

This implied micro-BOM would:

 - be generated at Quarkus build time, possibly by the same tooling which generates our quarkus-extension.properties

 - be checked for alignment with the platform, so to ensure different micro-BOM don't conflict with each other

The huge benefit would be to not need

a) the incidental complexity of managing intermediate bom files: most users couldn't care less (however they don't want the associated problems with the current approach)

b) to discuss what's the right level of granularity of each partial BOM (the right level of granularity is one for each extension)

I suppose we might still want a mega-BOM which lists all of our extensions for Maven users so that they don't go matching extensions meant for different Quarkus builds, but this would be unnecessary for Gradle users as the Gradle plugin could enforce extension alignment in other ways. Ideally it could be made unnecessary for Maven users as well, but I guess we could discuss that another time.

Additionally I suppose there might be some specific situations in which we might want to allow divergence between selected micro-BOM at cost of explicitly marking them as not compatible with each other; this could be useful for example to release an experimental extension introducing Hibernate ORM 6 as a possible alternative to the stable one which brings ORM 5. Overall I think it would be great to have such additional flexibility to allow for more experimentation and avoid us getting stuck because of self-imposed, unnecessary alignment requirements.

Thanks,

Sanne

--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAJ97idGvKGK8O83fpRO6pJRbQ4GsTv8sEkVFvYvBPDOFU7wj6g%40mail.gmail.com.

[Max Rydahl Andersen]

Sanne,

If I understand your suggestion right the perfect micro-bom would just be what each extension depends on - basically
if we could have every pom in the platform align all its dependencies we would all be good, correct?

And as that would require quite a massive alignment it isn't feasible thus instead make it so you generate a
bom somewhat dynamically based on what extensions your project actually uses ?

I see that as a possible future improvement to offer for those who really need it; but
I think having different default set of boms in play wether you use maven or gradle could end up being
a support and testing nightmare ?

I do think Alexey had a prototype for something similar for maven case but that would require use of manual configured maven extensions unfortunately.

No matter what - we do need to look into what the right granularity and version scheme are for the
"stacks".

/max

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAFm4XO02P4Dhu5Gxsp7DuVQO7mg-SHfqM%3Df4XwqsxAQwkHj_Dg%40mail.gmail.com.

/max
https://xam.dk/about

[Alexey Loubyansky]

On Wed, May 5, 2021 at 12:35 PM Sanne Grinovero <sa...@hibernate.org> wrote:

Hi Alexey,

big +1 from me to challenge the status quo, the huge BOM is very problematic and I love you're exploring this.

I do think you're on the right track conceptually, but I wonder if we could completely hide the notion of the "multiple fine-grained BOMs" you're proposing?

Ideally each extension should have an implicit BOM, which is automatically applied when the extension is imported.

I actually did some research on that last summer, i think.

This implied micro-BOM would:

 - be generated at Quarkus build time, possibly by the same tooling which generates our quarkus-extension.properties

 - be checked for alignment with the platform, so to ensure different micro-BOM don't conflict with each other

That's the tricky point. Which platform would that be? Basically, an extension-specific BOM generated at extension build time is not going to be useful for extensions that will be integrated into a platform (for which the dependency constraints should be aligned), assuming we allow extensions to have their own release cycle (i.e. core, camel, kogito, etc). One extension may cause adjustments of the constraints of another extension. Unless all the platform extensions are released together (basically as a single project), the BOM generated at extension build time will have to be regenerated at platform release time or a new extension release will be required. Then, I supposed, the same extension release may be included in different versions of the same platform. Different versions of the same platform may have different constraint alignments, with all the consequences.

In addition to that, a platform, among other things, represents a dev stack that was composed with some primary use-cases in mind. So, there could be multiple platforms (specialized dev stacks) and the same extension may appear in more than one platform with different dep constraint requirements, with all the consequences.

A side note, IMO, at some point, we should stop thinking of a platform as a 'universe'. At the time, I actually proposed to change the original artifactId from 'quarkus-platform-bom' to 'quarkus-universe-bom' because it was clear to me the BOM we were creating wasn't strictly a platform but just a phase in the evolution of the ecosystem and tools around it. It was important to create the basic notion of the platform but also have a way to contribute extensions from outside quarkus core. So I didn't want the term 'platform' to be diluted (at least that artifact id) and keep it for the future, just in case.

So, I am thinking some parts of the extension dependency constraints, JSON metadata and even some config options should be defined in the platform (dev stack) the extension is a part of instead of being fixed at the extension build time.

The huge benefit would be to not need

a) the incidental complexity of managing intermediate bom files: most users couldn't care less (however they don't want the associated problems with the current approach)

b) to discuss what's the right level of granularity of each partial BOM (the right level of granularity is one for each extension)

I suppose we might still want a mega-BOM which lists all of our extensions for Maven users so that they don't go matching extensions meant for different Quarkus builds, but this would be unnecessary for Gradle users as the Gradle plugin could enforce extension alignment in other ways. Ideally it could be made unnecessary for Maven users as well, but I guess we could discuss that another time.

Additionally I suppose there might be some specific situations in which we might want to allow divergence between selected micro-BOM at cost of explicitly marking them as not compatible with each other; this could be useful for example to release an experimental extension introducing Hibernate ORM 6 as a possible alternative to the stable one which brings ORM 5. Overall I think it would be great to have such additional flexibility to allow for more experimentation and avoid us getting stuck because of self-imposed, unnecessary alignment requirements.

I think the proposal is a step in that direction. I still like the idea of extension-specific BOMs and it should be researched further. The UX, in particular. We definitely don't want to see a BOM import per extension in POMs. Doing it "behind the scenes" is something to be researched, making sure the IDEs will still work too.

[Sanne Grinovero]

On Wed, 5 May 2021 at 12:57, Max Rydahl Andersen <mand...@redhat.com> wrote:

Sanne,

If I understand your suggestion right the perfect micro-bom would just be what each extension depends on - basically
if we could have every pom in the platform align all its dependencies we would all be good, correct?

That's what we already do today, no?

With this proposal we're just taking it a step further: don't impose a dependencyManagement section on end users which "manages" dependencies of extensions which are not in use. 

And as that would require quite a massive alignment it isn't feasible thus instead make it so you generate a
bom somewhat dynamically based on what extensions your project actually uses ?

Sure but you need to specify the "somewhat dynamically", the devil is in the details here. You probably don't want it to behave like Maven would without any BOM - although it would do a fairly good job, it would forfait the exact match with the versions we've been testing the releases with which is IMO important - even more so when native-image gets into the picture.

So it seems we can logically infer we need this specified as to manage the dependencies to have an exact match with how Quarkus was built (pretty much the same as referring to our current mega-bom), except we only want to manage the dependencies of the extensions applied to the current project.

In that sense I agree, we should "dynamically" being able to assemble such a BOM by composing the BOM fragments as provided by each extension. They still have to match the original universe BOM though - generally speaking, but possibly in a more flexible way like I suggested we could make controlled exceptions when you have GoodReasonsTM.

I see that as a possible future improvement to offer for those who really need it; but
I think having different default set of boms in play wether you use maven or gradle could end up being
a support and testing nightmare ?

Agreed but that's were Alexeys' tooling can assert consistency. 

I do think Alexey had a prototype for something similar for maven case but that would require use of manual configured maven extensions unfortunately.

No matter what - we do need to look into what the right granularity and version scheme are for the
"stacks".

Well no that's exactly my point. Introducing "stacks" is a huge complexity barrier to end users IF it has impact on dependencies in unpredictable ways, and I can guarantee you that we won't get the granularity of "stacks" right at first shot - so it will also be a somewhat-in-flux fluid grouping, making it just more of a maintenance and compatibility nightmare when such stacks are adjusted and this suddenly widens/shrinks the net of managed dependencies.

If we manage to conflate a 1:1 relation between stacks and extensions, then you can avoid encumbering our users with the need to produce a new mental model of how this all works as people are already familiar with the notion and impact of adding dependencies to their tree.

N.B. you can still introduce "stacks" as a notion to help people getting started - but they simply don't have to play a role with the BOMs and universal alignment so let's not conflate the two ongoing proposals and projects. This would also allow you to adjust the granularity and exact content of each stack without breaking the effective BOM definitions as side-effects.

Thanks

[Sanne Grinovero]

On Wed, 5 May 2021 at 13:02, Alexey Loubyansky <alexey.l...@redhat.com> wrote:

On Wed, May 5, 2021 at 12:35 PM Sanne Grinovero <sa...@hibernate.org> wrote:

Hi Alexey,

big +1 from me to challenge the status quo, the huge BOM is very problematic and I love you're exploring this.

I do think you're on the right track conceptually, but I wonder if we could completely hide the notion of the "multiple fine-grained BOMs" you're proposing?

Ideally each extension should have an implicit BOM, which is automatically applied when the extension is imported.

I actually did some research on that last summer, i think.

This implied micro-BOM would:

 - be generated at Quarkus build time, possibly by the same tooling which generates our quarkus-extension.properties

 - be checked for alignment with the platform, so to ensure different micro-BOM don't conflict with each other

That's the tricky point. Which platform would that be? Basically, an extension-specific BOM generated at extension build time is not going to be useful for extensions that will be integrated into a platform (for which the dependency constraints should be aligned), assuming we allow extensions to have their own release cycle (i.e. core, camel, kogito, etc). One extension may cause adjustments of the constraints of another extension. Unless all the platform extensions are released together (basically as a single project), the BOM generated at extension build time will have to be regenerated at platform release time or a new extension release will be required. Then, I supposed, the same extension release may be included in different versions of the same platform. Different versions of the same platform may have different constraint alignments, with all the consequences.

In addition to that, a platform, among other things, represents a dev stack that was composed with some primary use-cases in mind. So, there could be multiple platforms (specialized dev stacks) and the same extension may appear in more than one platform with different dep constraint requirements, with all the consequences.

Can't you do it the same as today? It seems to me you're introducing micro-BOM as fragments of the current BOM, so a reasonable starting point is to still have the current BOM around and use it to generate the fragments.

This will imply that the extensions with their own release cycle still still have to adhere with the versions as defined by the core. I totally understand this doesn't solve all problems and you have many other plans in mind, but this is a small step in the right direction.

A side note, IMO, at some point, we should stop thinking of a platform as a 'universe'. At the time, I actually proposed to change the original artifactId from 'quarkus-platform-bom' to 'quarkus-universe-bom' because it was clear to me the BOM we were creating wasn't strictly a platform but just a phase in the evolution of the ecosystem and tools around it. It was important to create the basic notion of the platform but also have a way to contribute extensions from outside quarkus core. So I didn't want the term 'platform' to be diluted (at least that artifact id) and keep it for the future, just in case.

Right, the 'universe' won't scale but that's a different problem - it's still a good start to not impose alignment on thousands of libraries we don't need aligned.

 

So, I am thinking some parts of the extension dependency constraints, JSON metadata and even some config options should be defined in the platform (dev stack) the extension is a part of instead of being fixed at the extension build time.

The huge benefit would be to not need

a) the incidental complexity of managing intermediate bom files: most users couldn't care less (however they don't want the associated problems with the current approach)

b) to discuss what's the right level of granularity of each partial BOM (the right level of granularity is one for each extension)

I suppose we might still want a mega-BOM which lists all of our extensions for Maven users so that they don't go matching extensions meant for different Quarkus builds, but this would be unnecessary for Gradle users as the Gradle plugin could enforce extension alignment in other ways. Ideally it could be made unnecessary for Maven users as well, but I guess we could discuss that another time.

Additionally I suppose there might be some specific situations in which we might want to allow divergence between selected micro-BOM at cost of explicitly marking them as not compatible with each other; this could be useful for example to release an experimental extension introducing Hibernate ORM 6 as a possible alternative to the stable one which brings ORM 5. Overall I think it would be great to have such additional flexibility to allow for more experimentation and avoid us getting stuck because of self-imposed, unnecessary alignment requirements.

I think the proposal is a step in that direction. I still like the idea of extension-specific BOMs and it should be researched further. The UX, in particular. We definitely don't want to see a BOM import per extension in POMs. Doing it "behind the scenes" is something to be researched, making sure the IDEs will still work too.

Yes exactly, this is very much your proposal to which I'm suggesting only minimal changes to how you compute the BOM fragments: automatically, from our extension's dependencies - this should help with correctness too as we'd define exactly what we tested with.

BTW I think it would be totally fine to see a "BOM import per extension" in the POMs - sure it's verbose but POMs are verbose, and we have the tools to generate these.

On the upside, IDEs will be able to present the user with the best matching representation of the project (I still wish we could explore the behind the scenes idea but I think that could wait and probably needs some refinement)

Thanks

[Alexey Loubyansky]

On Thu, May 6, 2021 at 12:07 PM Sanne Grinovero <sa...@hibernate.org> wrote:

On Wed, 5 May 2021 at 13:02, Alexey Loubyansky <alexey.l...@redhat.com> wrote:

On Wed, May 5, 2021 at 12:35 PM Sanne Grinovero <sa...@hibernate.org> wrote:

Hi Alexey,

big +1 from me to challenge the status quo, the huge BOM is very problematic and I love you're exploring this.

I do think you're on the right track conceptually, but I wonder if we could completely hide the notion of the "multiple fine-grained BOMs" you're proposing?

Ideally each extension should have an implicit BOM, which is automatically applied when the extension is imported.

I actually did some research on that last summer, i think.

This implied micro-BOM would:

 - be generated at Quarkus build time, possibly by the same tooling which generates our quarkus-extension.properties

 - be checked for alignment with the platform, so to ensure different micro-BOM don't conflict with each other

That's the tricky point. Which platform would that be? Basically, an extension-specific BOM generated at extension build time is not going to be useful for extensions that will be integrated into a platform (for which the dependency constraints should be aligned), assuming we allow extensions to have their own release cycle (i.e. core, camel, kogito, etc). One extension may cause adjustments of the constraints of another extension. Unless all the platform extensions are released together (basically as a single project), the BOM generated at extension build time will have to be regenerated at platform release time or a new extension release will be required. Then, I supposed, the same extension release may be included in different versions of the same platform. Different versions of the same platform may have different constraint alignments, with all the consequences.

In addition to that, a platform, among other things, represents a dev stack that was composed with some primary use-cases in mind. So, there could be multiple platforms (specialized dev stacks) and the same extension may appear in more than one platform with different dep constraint requirements, with all the consequences.

Can't you do it the same as today? It seems to me you're introducing micro-BOM as fragments of the current BOM, so a reasonable starting point is to still have the current BOM around and use it to generate the fragments.

Right, that's how it's done.

This will imply that the extensions with their own release cycle still still have to adhere with the versions as defined by the core. I totally understand this doesn't solve all problems and you have many other plans in mind, but this is a small step in the right direction.

When a new version of core is released, the extension BOM may need to be re-generated, which implies releasing a new version of the extension, which could be avoided in case the extension BOM was not generated at extension build time.

A more interesting question is how would common dependencies of, e.g., Camel and Kogito be aligned? To make sure they can be used in the same project.

 

A side note, IMO, at some point, we should stop thinking of a platform as a 'universe'. At the time, I actually proposed to change the original artifactId from 'quarkus-platform-bom' to 'quarkus-universe-bom' because it was clear to me the BOM we were creating wasn't strictly a platform but just a phase in the evolution of the ecosystem and tools around it. It was important to create the basic notion of the platform but also have a way to contribute extensions from outside quarkus core. So I didn't want the term 'platform' to be diluted (at least that artifact id) and keep it for the future, just in case.

Right, the 'universe' won't scale but that's a different problem - it's still a good start to not impose alignment on thousands of libraries we don't need aligned.

 

So, I am thinking some parts of the extension dependency constraints, JSON metadata and even some config options should be defined in the platform (dev stack) the extension is a part of instead of being fixed at the extension build time.

The huge benefit would be to not need

a) the incidental complexity of managing intermediate bom files: most users couldn't care less (however they don't want the associated problems with the current approach)

b) to discuss what's the right level of granularity of each partial BOM (the right level of granularity is one for each extension)

I suppose we might still want a mega-BOM which lists all of our extensions for Maven users so that they don't go matching extensions meant for different Quarkus builds, but this would be unnecessary for Gradle users as the Gradle plugin could enforce extension alignment in other ways. Ideally it could be made unnecessary for Maven users as well, but I guess we could discuss that another time.

Additionally I suppose there might be some specific situations in which we might want to allow divergence between selected micro-BOM at cost of explicitly marking them as not compatible with each other; this could be useful for example to release an experimental extension introducing Hibernate ORM 6 as a possible alternative to the stable one which brings ORM 5. Overall I think it would be great to have such additional flexibility to allow for more experimentation and avoid us getting stuck because of self-imposed, unnecessary alignment requirements.

I think the proposal is a step in that direction. I still like the idea of extension-specific BOMs and it should be researched further. The UX, in particular. We definitely don't want to see a BOM import per extension in POMs. Doing it "behind the scenes" is something to be researched, making sure the IDEs will still work too.

Yes exactly, this is very much your proposal to which I'm suggesting only minimal changes to how you compute the BOM fragments: automatically, from our extension's dependencies - this should help with correctness too as we'd define exactly what we tested with.

Unless all the extensions that are supposed to be compatible belong to the same project and are released together, each extension-specific BOM will be computed in isolation. It would be possible to run a CI against some set of extensions (basically our platform TS) to verify if it's still compatible. But resolving conflicts by dependency alignments would basically translate into releases of extensions from multiple repos. Which will be hard to manage.

BTW I think it would be totally fine to see a "BOM import per extension" in the POMs - sure it's verbose but POMs are verbose, and we have the tools to generate these.

This is where I disagree. We've already received a lot of hate for 4 properties in our generated BOMs. A BOM import per extension dependency will definitely not be appreciated. A lot of people prefer writing/editing their BOMs manually and even if not, it'll simply be horrible, imo.

Extension-specific BOMs can be generated easily in my PoC, btw. I am not proposing it exactly for the UX reasons.

 

On the upside, IDEs will be able to present the user with the best matching representation of the project (I still wish we could explore the behind the scenes idea but I think that could wait and probably needs some refinement)

Thanks

--

You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAFm4XO0FqUgnU6dXqEpW7MR93G%3DmAMP7Dbwb6xPrBQnVjsQDyw%40mail.gmail.com.

[Sanne Grinovero]

Ok, thanks for clarifying. Personally I don't think it's that bad as it's explicit and a well known pattern, but I understand you're under pressure. Are you sure the issue with the current properties isn't that they are confusing and apparently redundant? I don't think people are yelling because of the space they take in the file.

For example, when I have to debug a reproducer and have to switch to the current snapshot, that I also have to remember changing parent BOMs - that's the kind of stuff I see people most confused about.

[Alexey Loubyansky]

I won't go into clarifying their purpose to not start another debate on that. But their original purpose is diminished with this proposal, so I personally would remove them, perhaps keeping only the "quarkus.version".

[Max Rydahl Andersen]

On 6 May 2021, at 11:51, Sanne Grinovero wrote:

> On Wed, 5 May 2021 at 12:57, Max Rydahl Andersen <mand...@redhat.com>
> wrote:
>
>> Sanne,
>>
>> If I understand your suggestion right the perfect micro-bom would
>> just be
>> what each extension depends on - basically
>> if we could have every pom in the platform align all its dependencies
>> we
>> would all be good, correct?
>>
> That's what we already do today, no?

No, if you remove the quarkus bom and just use extensions directly you
will get
a different resolved dependency set ( no matter if you use gradle or
maven ).

this is particular true for out of core extensions like kogito or camel
but
I've seen it on i.e. resteasy extension where the transitive
dependencies of
resteasy is different than when constrained by quarks platform bom.

> With this proposal we're just taking it a step further: don't impose a
> dependencyManagement section on end users which "manages" dependencies
> of
> extensions which are not in use.
>
>> And as that would require quite a massive alignment it isn't feasible
>> thus
>> instead make it so you generate a
>> bom somewhat dynamically based on what extensions your project
>> actually
>> uses ?
>>
> Sure but you need to specify the "somewhat dynamically", the devil is
> in
> the details here. You probably don't want it to behave like Maven
> would
> without any BOM - although it would do a fairly good job, it would
> forfait
> the exact match with the versions we've been testing the releases with
> which is IMO important - even more so when native-image gets into the
> picture.

today we do test with quarkus bom and platform bom - not with the
natural derived dependency set of every extension.

> So it seems we can logically infer we need this specified as to manage
> the
> dependencies to have an exact match with how Quarkus was built (pretty
> much
> the same as referring to our current mega-bom), except we only want to
> manage the dependencies of the extensions applied to the current
> project.
>
> In that sense I agree, we should "dynamically" being able to assemble
> such
> a BOM by composing the BOM fragments as provided by each extension.
> They
> still have to match the original universe BOM though - generally
> speaking,
> but possibly in a more flexible way like I suggested we could make
> controlled exceptions when you have GoodReasonsTM.


> I see that as a possible future improvement to offer for those who
> really
>> need it; but

>> I think having *different* default set of boms in play wether you use

>> maven or gradle could end up being
>> a support and testing nightmare ?
>>
> Agreed but that's were Alexeys' tooling can assert consistency.

true, but I think that would be a 2+ feature to look at as for now it
would
require use of maven extensions to make it consistent.

>> I do think Alexey had a prototype for something similar for maven
>> case but
>> that would require use of manual configured maven extensions
>> unfortunately.
>>
>> No matter what - we do need to look into what the right granularity
>> and
>> version scheme are for the
>> "stacks".
>>

The 'platform stacks' granularity I'm thinking about is that it plays a
role for how we actually coordinate build of the platform.

> Well no that's exactly my point. Introducing "stacks" is a huge
> complexity
> barrier to end users IF it has impact on dependencies in unpredictable
> ways, and I can guarantee you that we won't get the granularity of
> "stacks"
> right at first shot - so it will also be a somewhat-in-flux fluid
> grouping,
> making it just more of a maintenance and compatibility nightmare when
> such
> stacks are adjusted and this suddenly widens/shrinks the net of
> managed
> dependencies.

Yes, but how do we get from where we are now to the world of fluid
dynamic bom generation ?

I don't think we can do that in one big move and expect it to be right?

> If we manage to conflate a 1:1 relation between stacks and extensions,
> then
> you can avoid encumbering our users with the need to produce a new
> mental
> model of how this all works as people are already familiar with the
> notion
> and impact of adding dependencies to their tree.

> N.B. you can still introduce "stacks" as a notion to help people
> getting
> started - but they simply don't have to play a role with the BOMs and
> universal alignment so let's not conflate the two ongoing proposals
> and
> projects. This would also allow you to adjust the granularity and
> exact
> content of each stack without breaking the effective BOM definitions
> as
> side-effects.

Not sure I follow your N.B. suggestion here ? can you give example ?

/max

>>> <https://groups.google.com/d/msgid/quarkus-dev/CAJ97idGvKGK8O83fpRO6pJRbQ4GsTv8sEkVFvYvBPDOFU7wj6g%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>> .

>>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups
>> "Quarkus Development mailing list" group.
>> To unsubscribe from this group and stop receiving emails from it,
>> send an
>> email to quarkus-dev...@googlegroups.com.
>> To view this discussion on the web visit

>> https://groups.google.com/d/msgid/quarkus-dev/CAFm4XO02P4Dhu5Gxsp7DuVQO7mg-SHfqM%3Df4XwqsxAQwkHj_Dg%40mail.gmail.com
>> <https://groups.google.com/d/msgid/quarkus-dev/CAFm4XO02P4Dhu5Gxsp7DuVQO7mg-SHfqM%3Df4XwqsxAQwkHj_Dg%40mail.gmail.com?utm_medium=email&utm_source=footer>
>> .
>>
>> /max
>> https://xam.dk/about
>>


/max
https://xam.dk/about

[Sanne Grinovero]

On Fri, 7 May 2021 at 00:00, Max Rydahl Andersen <mand...@redhat.com> wrote:

On 6 May 2021, at 11:51, Sanne Grinovero wrote:

> On Wed, 5 May 2021 at 12:57, Max Rydahl Andersen <mand...@redhat.com>
> wrote:
>
>> Sanne,
>>
>> If I understand your suggestion right the perfect micro-bom would
>> just be
>> what each extension depends on - basically
>> if we could have every pom in the platform align all its dependencies
>> we
>> would all be good, correct?
>>
> That's what we already do today, no?

No, if you remove the quarkus bom and just use extensions directly you
will get
a different resolved dependency set ( no matter if you use gradle or
maven ).

I'm really struggling to follow you Max, I never suggested to get rid of BOMs;

Alexey suggested to split BOMs in smaller ones and I am aligned with that,

I'm just highlighting that at *implementation level* the right level of granularity

of which dependencies we should be managing is 1:1 aligned with extensions,

rather than "stacks".

You challenged me with alleged difficulties about which versions these BOMs

should have, to which I don't have an answer for except pointing out that we

can keep producing these BOMs in the same way as we do today: sourcing

the versions from a unique manifest file which enforces dependencies across

all artifacts in the same repository.

Sure this isn't scaling too well for us and once we have separation of BOMs

in place we can work on allowing ad-hoc, controlled exceptions to the rule,

but we can at least keep producing BOMs so to implement the dependency

alignment and prevent the issues such as the RESTEasy one you mention,

but also - more importantly - impose usage with strict adherence to the versions

we run integration tests with.

That is exactly what we do today, I agree it's not perfect but I'm not making any

suggestions about changing this - I'm just welcoming the proposal of splitting

as we will need the additional flexibility that this will make available.

In my first paragraph today I highlight "implementation" so to stress that I'm not

expressing an opinion against introducing the notion of "stacks" at user level,

but if stacks are a notion to help users getting started I'm suggesting that they

should stick at being a consistent aggregation of extensions as a concept,

they don't necessarily have to be implemented as a single BOM.

Alexey then pointed out that people would need to import many BOMs,

I'm not 100% persuaded that this would be a real problem but I can trust him

on this and appreciate it's a factor that needs taking into account.

Maybe ideally we could even produce the micro-BOMs like I suggested and

define stacks as nothing more than a BOM which aggregates multiple micro-BOMs?

This way we'd have the option for users to go micro and a clean definition of "stacks",

defined purely as user-friendly sugar on top of a sound base layer.

The benefit I see in the 1:1 alignment with extensions is that:

 - people who prefer the minimal option over stacks have such an option

 - other platforms can define their own "stacks" by mix&match of core extensions

 - end users to run with dependencies perfectly aligned to how we run integration tests

    -- OR our tooling being able to warn against an inconsistency, useful for native.

HTH

[Max Rydahl Andersen]

On 7 May 2021, at 11:29, Sanne Grinovero wrote:

sorry for missing this email! albeit late comments are below.

On Fri, 7 May 2021 at 00:00, Max Rydahl Andersen mand...@redhat.com
wrote:

On 6 May 2021, at 11:51, Sanne Grinovero wrote:

On Wed, 5 May 2021 at 12:57, Max Rydahl Andersen mand...@redhat.com
wrote:

Sanne,

If I understand your suggestion right the perfect micro-bom would
just be
what each extension depends on - basically
if we could have every pom in the platform align all its dependencies
we
would all be good, correct?

That's what we already do today, no?

No, if you remove the quarkus bom and just use extensions directly you
will get
a different resolved dependency set ( no matter if you use gradle or
maven ).

I'm really struggling to follow you Max, I never suggested to get rid of
BOMs;
Alexey suggested to split BOMs in smaller ones and I am aligned with that,

I'm just highlighting that at implementation level the right level of

granularity
of which dependencies we should be managing is 1:1 aligned with extensions,
rather than "stacks".

Ack, and I understand the sentiment here - but it does mean alot of at least perceived
duplication in maven pom.xml just to use an extension...

You challenged me with alleged difficulties about which versions these BOMs
should have, to which I don't have an answer for except pointing out that we
can keep producing these BOMs in the same way as we do today: sourcing
the versions from a unique manifest file which enforces dependencies across
all artifacts in the same repository.

Sure this isn't scaling too well for us and once we have separation of BOMs
in place we can work on allowing ad-hoc, controlled exceptions to the rule,
but we can at least keep producing BOMs so to implement the dependency
alignment and prevent the issues such as the RESTEasy one you mention,
but also - more importantly - impose usage with strict adherence to the
versions
we run integration tests with.

That is exactly what we do today, I agree it's not perfect but I'm not
making any
suggestions about changing this - I'm just welcoming the proposal of
splitting
as we will need the additional flexibility that this will make available.

I think we are agreeing :) just that instead of going from 1 bom to 150+ boms
we are working towards having a system in place that lets us actually get there eventually
but starting out with "stacks" matching where the team/functional boundaries are as that at least
lets us get started.

In my first paragraph today I highlight "implementation" so to stress that
I'm not
expressing an opinion against introducing the notion of "stacks" at user
level,
but if stacks are a notion to help users getting started I'm suggesting
that they
should stick at being a consistent aggregation of extensions as a concept,
they don't necessarily have to be implemented as a single BOM.

Alexey then pointed out that people would need to import many BOMs,
I'm not 100% persuaded that this would be a real problem but I can trust him
on this and appreciate it's a factor that needs taking into account.

Maybe ideally we could even produce the micro-BOMs like I suggested and
define stacks as nothing more than a BOM which aggregates multiple
micro-BOMs?
This way we'd have the option for users to go micro and a clean definition
of "stacks",
defined purely as user-friendly sugar on top of a sound base layer.

I definitely can see the value in exploring that option!

Alexey - how hard would this actually be to do as an experiment in the current bom generator ?
Basically iterate every extension can generate a matching io.quarkus.platform.ext:artifact-id.bom?

The benefit I see in the 1:1 alignment with extensions is that:

• people who prefer the minimal option over stacks have such an option

• other platforms can define their own "stacks" by mix&match of core

extensions

• end users to run with dependencies perfectly aligned to how we run

integration tests
-- OR our tooling being able to warn against an inconsistency, useful
for native.

Agreed; we'll though also want our project creation to pick one of the approaches.

in short - I think we are on the right path to explore this!

/max

HTH

need it; but
I think having different default set of boms in play wether you use

Thanks

• be generated at Quarkus build time, possibly by the same tooling

which
generates our quarkus-extension.properties

• be checked for alignment with the platform, so to ensure different

1. MOTIVATION

2. PROPOSAL

platform release may imply updating more than one BOM version. The

dev
tools will be able to check for and list the available updates, and
also
apply them, if requested.
Users who prefer editing project configs manually would be able to
use
the tools to list the available updates and apply them manually.

Manual project config maintenance becomes more error prone with this
model. The good news is that, at app build time, we would be able to
check
whether the imported platform BOMs belong to the same platform stack
release and are a tested combination. If it’s not the case we
would display
a clear error/warning message indicating which BOM versions were
found to
be misaligned and how the versions should be adjusted to have a
properly
aligned stack.

3. APPLICATION POM EXAMPLES

4. PoC

/max
https://xam.dk/about

[Alexey Loubyansky]

As mentioned in this thread previously, it is possible to generate **in the context of the platform** extension-specific BOMs. I've actually done that in a couple of prototypes. It's not difficult but also not as trivial as you might think, unless we are ok compromising some dev use-cases by limiting the BOMs to the actual build and runtime classpaths of each extension.

For what we are doing today though, composing the member BOMs by importing extension-specific BOMs will have a negative effect on app build time and in general will actually complicate the impl of the BOM generator. So I wouldn't do that at this point. But once we sort things out with the changes we are introducing, generating extension-specific BOMs and releasing them along with the platform will be a good way to evaluate the approach. It would be similar to what we are doing now with the universe and member-specific BOMs.