Jackson version alignment with Gradle 6

[Jendrik Johannes]

Hello from Gradle,

Maybe the following blog post about version alignment by using a BOM (with Gradle 6), is of interest for some who use Jackson in Gradle builds. It uses Jackson, and the Jackson BOM, as an example:

https://blog.gradle.org/alignment-with-gradle-module-metadata

In a nutshell, Gradle 6 allows you to publish dependencies on platforms (BOMs) together with each module. By this mechanism, any Jackson module you use automatically pulls in the corresponding BOM, which in turn aligns all other Jackson modules on the dependency graph to the same version. This avoids misaligned versions if different modules are added through transitive dependencies. The example is described in more detail in the post.

For this to work, the modules need to publish Gradle Module Metadata (GMM) in addition to their POM files. This new metadata format was introduced recently and is now fully supported in Gradle 6:

https://docs.gradle.org/current/userguide/publishing_gradle_module_metadata.html

https://blog.gradle.org/gradle-metadata-1.0

Gradle also offers an API to add additional metadata that was not published. I created a small Gradle plugin which does that for all Jackson versions that have a BOM. You can use that in your Gradle build if you want to try what is described in the blog post:

https://plugins.gradle.org/plugin/de.jjohannes.missing-metadata-jackson

From what I see about how Jackson is published, I realize that it is not easy to actually publish this additional metadata for future versions. (Although, for the benefit of Gradle users, it would be a great thing!)

In any case, I want to offer support from the Gradle side if there should be interest in this topic from the Jackson developers. While using Gradle as build tool makes it easy to publish the additional metadata, it is also possible to find solutions with Maven builds. We are exploring that, for example, with Guava's build which has some other interesting use cases solvable with GMM (https://blog.gradle.org/guava). 

Regards,
Jendrik

[Tatu Saloranta]

On Mon, Feb 3, 2020 at 9:46 AM Jendrik Johannes <jen...@gradle.com> wrote:

Hello from Gradle,

Hi there!

 

Maybe the following blog post about version alignment by using a BOM (with Gradle 6), is of interest for some who use Jackson in Gradle builds. It uses Jackson, and the Jackson BOM, as an example:

https://blog.gradle.org/alignment-with-gradle-module-metadata

Very interesting. Will need read with thought.Thank you for sharing this.

In a nutshell, Gradle 6 allows you to publish dependencies on platforms (BOMs) together with each module. By this mechanism, any Jackson module you use automatically pulls in the corresponding BOM, which in turn aligns all other Jackson modules on the dependency graph to the same version. This avoids misaligned versions if different modules are added through transitive dependencies. The example is described in more detail in the post.

For this to work, the modules need to publish Gradle Module Metadata (GMM) in addition to their POM files. This new metadata format was introduced recently and is now fully supported in Gradle 6:

https://docs.gradle.org/current/userguide/publishing_gradle_module_metadata.html

https://blog.gradle.org/gradle-metadata-1.0

Gradle also offers an API to add additional metadata that was not published. I created a small Gradle plugin which does that for all Jackson versions that have a BOM. You can use that in your Gradle build if you want to try what is described in the blog post:

https://plugins.gradle.org/plugin/de.jjohannes.missing-metadata-jackson

From what I see about how Jackson is published, I realize that it is not easy to actually publish this additional metadata for future versions. (Although, for the benefit of Gradle users, it would be a great thing!)

In any case, I want to offer support from the Gradle side if there should be interest in this topic from the Jackson developers. While using Gradle as build tool makes it easy to publish the additional metadata, it is also possible to find solutions with Maven builds. We are exploring that, for example, with Guava's build which has some other interesting use cases solvable with GMM (https://blog.gradle.org/guava). 

Yes, I think this is an area where we would be interested in finding improvements.

It is probably true that on short term it may be challenging to provide metadata, but at the same time Jackson is producing some metadata using non-native tooling:

- OSGi metadata has been generated with Felix plug-in for years now (and mostly works, although project relies on OSGi users to report issues as tooling can only verify some problems)

- Java 9+ module info is generated using Moditect plug-in since JDK tooling only available on JDK 9 and later (which won't work too well for producing versions usable on Java 8 and earlier)

so perhaps there could be a way to include some more information even if build itself uses Maven.

-+ Tatu +-

 

Regards,
Jendrik

--
You received this message because you are subscribed to the Google Groups "jackson-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jackson-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jackson-user/20edbae4-a327-48b4-b056-079661663db2%40googlegroups.com.

[Jendrik Johannes]

Hi Tatu,

Thanks for your interest!

 

Maybe the following blog post about version alignment by using a BOM (with Gradle 6), is of interest for some who use Jackson in Gradle builds. It uses Jackson, and the Jackson BOM, as an example:

https://blog.gradle.org/alignment-with-gradle-module-metadata

Very interesting. Will need read with thought.Thank you for sharing this.

Let me know if you have any questions.

 

From what I see about how Jackson is published, I realize that it is not easy to actually publish this additional metadata for future versions. (Although, for the benefit of Gradle users, it would be a great thing!)

In any case, I want to offer support from the Gradle side if there should be interest in this topic from the Jackson developers. While using Gradle as build tool makes it easy to publish the additional metadata, it is also possible to find solutions with Maven builds. We are exploring that, for example, with Guava's build which has some other interesting use cases solvable with GMM (https://blog.gradle.org/guava). 

Yes, I think this is an area where we would be interested in finding improvements.

It is probably true that on short term it may be challenging to provide metadata, but at the same time Jackson is producing some metadata using non-native tooling:

- OSGi metadata has been generated with Felix plug-in for years now (and mostly works, although project relies on OSGi users to report issues as tooling can only verify some problems)

- Java 9+ module info is generated using Moditect plug-in since JDK tooling only available on JDK 9 and later (which won't work too well for producing versions usable on Java 8 and earlier)

so perhaps there could be a way to include some more information even if build itself uses Maven.

Good to know about the other metadata you publish. I wasn't aware of that. Technically, it is not a problem to publish Gradle Module Metadata (GMM) with Maven. The metadata file itself is following Maven naming conventions and can as such be treated as an additional artifact when publishing with Maven (I have done a poc using the "build-helper-maven-plugin" on the Guava build here). In that example, the GMM file is maintained manually in parallel to the POM.

In your case, it would probably be good to have a Maven plugin that generates, or enriches, a GMM file with the dependency information from the POMs. 

The main concern is conceptually with the Jackson release process. The approach described in the blog post essentially requires all components and BOM to be released together. Because the components refer to the BOM and the BOM refers to the components. If I understand the current release process correctly, the BOM is published in the end once all components were released individually. I don't know if that is something you would consider to do differently in the future?

A variation of the approach works BOM. Then, components declare what we call dependency constraints directly in their metadata.

For example, this could be done for the core components only (if they are released together):

jackson-core-2.10.2.module             { dependency-constraints { jackson-databind:2.10.2, jackson-annotations:2.10.2, } } 

jackson-databind-2.10.2-module      { dependency-constraints { jackson-core:2.10.2, jackson-annotations:2.10.2, } } 

jackson-annotations-2.10.2.module { dependency-constraints { jackson-core:2.10.2, jackson-databind:2.10.2, } } 

This means that if you select `2.10.2` for any of the three components, `2.10.2` will also be selected for the other two (through a constraint directly defined in the metadata).

Cheers,

Jendrik

[Tatu Saloranta]

On Thu, Feb 6, 2020 at 2:15 AM Jendrik Johannes <jen...@gradle.com> wrote:
>
> Hi Tatu,
>
> Thanks for your interest!
>
>>
>>>
>>> Maybe the following blog post about version alignment by using a BOM (with Gradle 6), is of interest for some who use Jackson in Gradle builds. It uses Jackson, and the Jackson BOM, as an example:
>>>
>>> https://blog.gradle.org/alignment-with-gradle-module-metadata
>>
>>
>> Very interesting. Will need read with thought.Thank you for sharing this.
>
>
> Let me know if you have any questions.
>
>>>
>>> From what I see about how Jackson is published, I realize that it is not easy to actually publish this additional metadata for future versions. (Although, for the benefit of Gradle users, it would be a great thing!)
>>>
>>> In any case, I want to offer support from the Gradle side if there should be interest in this topic from the Jackson developers. While using Gradle as build tool makes it easy to publish the additional metadata, it is also possible to find solutions with Maven builds. We are exploring that, for example, with Guava's build which has some other interesting use cases solvable with GMM (https://blog.gradle.org/guava).
>>>
>>>
>> Yes, I think this is an area where we would be interested in finding improvements.
>> It is probably true that on short term it may be challenging to provide metadata, but at the same time Jackson is producing some metadata using non-native tooling:
>>
>> - OSGi metadata has been generated with Felix plug-in for years now (and mostly works, although project relies on OSGi users to report issues as tooling can only verify some problems)
>> - Java 9+ module info is generated using Moditect plug-in since JDK tooling only available on JDK 9 and later (which won't work too well for producing versions usable on Java 8 and earlier)
>>
>> so perhaps there could be a way to include some more information even if build itself uses Maven.
>
>
> Good to know about the other metadata you publish. I wasn't aware of that. Technically, it is not a problem to publish Gradle Module Metadata (GMM) with Maven. The metadata file itself is following Maven naming conventions and can as such be treated as an additional artifact when publishing with Maven (I have done a poc using the "build-helper-maven-plugin" on the Guava build here). In that example, the GMM file is maintained manually in parallel to the POM.
>
> In your case, it would probably be good to have a Maven plugin that generates, or enriches, a GMM file with the dependency information from the POMs.

Yes.

> The main concern is conceptually with the Jackson release process. The approach described in the blog post essentially requires all components and BOM to be released together. Because the components refer to the BOM and the BOM refers to the components. If I understand the current release process correctly, the BOM is published in the end once all components were released individually. I don't know if that is something you would consider to do differently in the future?

Probably not; while unification of some of projects has been done (and
may be done in future), I don't think full mono-repo for all Jackson
components is the eventual goal.

Typically bom is actually release early in the process, since most
components (excluding jackson-annotations and jackson-core that do not
depend on any other jackson components) use version set from BOM (that
is, extend `jackson-base`, which imports or extends bom).
I don't know if this makes much difference.

> A variation of the approach works BOM. Then, components declare what we call dependency constraints directly in their metadata.
> For example, this could be done for the core components only (if they are released together):
>
> jackson-core-2.10.2.module { dependency-constraints { jackson-databind:2.10.2, jackson-annotations:2.10.2, } }
> jackson-databind-2.10.2-module { dependency-constraints { jackson-core:2.10.2, jackson-annotations:2.10.2, } }
> jackson-annotations-2.10.2.module { dependency-constraints { jackson-core:2.10.2, jackson-databind:2.10.2, } }
>
> This means that if you select `2.10.2` for any of the three components, `2.10.2` will also be selected for the other two (through a constraint directly defined in the metadata).

This could maybe work?

Jackson 3.0 will change versioning slightly, dropping patch from
annotations (there will only be 3.0, 3.1, unless some critical problem
were found requiring patch version), which might complicate this in
that annotations would only limit minor version of other components.

-+ Tatu +-

[Jendrik Johannes]

On Saturday, February 8, 2020 at 12:32:50 AM UTC+1, Tatu Saloranta wrote:

> In your case, it would probably be good to have a Maven plugin that generates, or enriches, a GMM file with the dependency information from the POMs.

Yes.

If we get to a conclusion about a doable approach below, I can have a look into providing this.

 

> The main concern is conceptually with the Jackson release process. The approach described in the blog post essentially requires all components and BOM to be released together. Because the components refer to the BOM and the BOM refers to the components. If I understand the current release process correctly, the BOM is published in the end once all components were released individually. I don't know if that is something you would consider to do differently in the future?

Probably not; while unification of some of projects has been done (and
may be done in future), I don't think full mono-repo for all Jackson
components is the eventual goal.

Typically bom is actually release early in the process, since most
components (excluding jackson-annotations and jackson-core that do not
depend on any other jackson components) use version set from BOM (that
is, extend `jackson-base`, which imports or extends bom).
I don't know if this makes much difference.

So how does that work exactly? (If there is documentation somewhere I missed, please point me to it.)

Lets take an arbitrary example - "jackson-jaxrs-json-provider".

When you published "2.10.2", did you publish the BOM before you published that module? Because the BOM contains the entry:

<dependency>

  <groupId>com.fasterxml.jackson.jaxrs</groupId>

  <artifactId>jackson-jaxrs-json-provider</artifactId>

  <version>${jackson.version.jaxrs}</version>

</dependency>

Which is invalid until "jackson-jaxrs-json-provider" is actually published. 

> A variation of the approach works BOM. Then, components declare what we call dependency constraints directly in their metadata.
> For example, this could be done for the core components only (if they are released together):
>
> jackson-core-2.10.2.module             { dependency-constraints { jackson-databind:2.10.2, jackson-annotations:2.10.2, } }
> jackson-databind-2.10.2-module      { dependency-constraints { jackson-core:2.10.2, jackson-annotations:2.10.2, } }
> jackson-annotations-2.10.2.module { dependency-constraints { jackson-core:2.10.2, jackson-databind:2.10.2, } }
>
> This means that if you select `2.10.2` for any of the three components, `2.10.2` will also be selected for the other two (through a constraint directly defined in the metadata).
> This could maybe work?

Jackson 3.0 will change versioning slightly, dropping patch from
annotations (there will only be 3.0, 3.1, unless some critical problem
were found requiring patch version), which might complicate this in
that annotations would only limit minor version of other components.

This is not be a problem. The versions do not have to be the same. As long as we know "what fits together". 

Jendrik 

[Tatu Saloranta]

On Mon, Feb 10, 2020 at 12:37 AM Jendrik Johannes <jen...@gradle.com> wrote:
>
>
> On Saturday, February 8, 2020 at 12:32:50 AM UTC+1, Tatu Saloranta wrote:
>>
>> > In your case, it would probably be good to have a Maven plugin that generates, or enriches, a GMM file with the dependency information from the POMs.
>>
>> Yes.
>
>
> If we get to a conclusion about a doable approach below, I can have a look into providing this.
>
>>
>>
>> > The main concern is conceptually with the Jackson release process. The approach described in the blog post essentially requires all components and BOM to be released together. Because the components refer to the BOM and the BOM refers to the components. If I understand the current release process correctly, the BOM is published in the end once all components were released individually. I don't know if that is something you would consider to do differently in the future?
>>
>> Probably not; while unification of some of projects has been done (and
>> may be done in future), I don't think full mono-repo for all Jackson
>> components is the eventual goal.
>>
>> Typically bom is actually release early in the process, since most
>> components (excluding jackson-annotations and jackson-core that do not
>> depend on any other jackson components) use version set from BOM (that
>> is, extend `jackson-base`, which imports or extends bom).
>> I don't know if this makes much difference.
>
>
> So how does that work exactly? (If there is documentation somewhere I missed, please point me to it.)
> Lets take an arbitrary example - "jackson-jaxrs-json-provider".
> When you published "2.10.2", did you publish the BOM before you published that module? Because the BOM contains the entry:
>
> <dependency>
> <groupId>com.fasterxml.jackson.jaxrs</groupId>
> <artifactId>jackson-jaxrs-json-provider</artifactId>
> <version>${jackson.version.jaxrs}</version>
> </dependency>
>
> Which is invalid until "jackson-jaxrs-json-provider" is actually published.

Correct in the sense that published BOM is not usable until referenced
versions are accessible. But Maven publishing itself does not care nor
validate it.
So there is a window of opportunity for inconsistent usage.

At the same time, this does make it possible for Jackson artifacts to
refer to jackson-bom (via jackson-base that extends it, published from
same github repo).
So 2.10.3 version `jackson-datatype-guava`, for example, can use
`jackson-base` 2.10.3 which extends 2.10.3 of `jackson-bom`, which
then provides 2.10.3 version for things it needs (annotations, core,
databind).

While not exactly minimal amount of work (artifacts do need to update
`jackson-base` dependency version on publishing, in addition to their
own version), this is somewhat better than the way individual
dependencies were increased in the past.
For what that's worth.

>> > A variation of the approach works BOM. Then, components declare what we call dependency constraints directly in their metadata.
>> > For example, this could be done for the core components only (if they are released together):
>> >
>> > jackson-core-2.10.2.module { dependency-constraints { jackson-databind:2.10.2, jackson-annotations:2.10.2, } }
>> > jackson-databind-2.10.2-module { dependency-constraints { jackson-core:2.10.2, jackson-annotations:2.10.2, } }
>> > jackson-annotations-2.10.2.module { dependency-constraints { jackson-core:2.10.2, jackson-databind:2.10.2, } }
>> >
>> > This means that if you select `2.10.2` for any of the three components, `2.10.2` will also be selected for the other two (through a constraint directly defined in the metadata).
>> > This could maybe work?
>>
>> Jackson 3.0 will change versioning slightly, dropping patch from
>> annotations (there will only be 3.0, 3.1, unless some critical problem
>> were found requiring patch version), which might complicate this in
>> that annotations would only limit minor version of other components.
>
>
> This is not be a problem. The versions do not have to be the same. As long as we know "what fits together".

Right, good! Just thought I'd mention it, did not think it would be a blocker.

-+ Tatu +-

>
> Jendrik
>
> --
> You received this message because you are subscribed to the Google Groups "jackson-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to jackson-user...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/jackson-user/edef900a-fa2e-4413-9b03-82461f106e90%40googlegroups.com.

[Jendrik Johannes]

Interesting. Thanks for clarifying. So this means that the approach from the blog post would work as the BOM is published *before* the components.

Everything would work as expected once all componentss got published for a version. The only 'interesting' situation is while not all componentss are published yet. Then Gradle could end up looking for components that do not exist yet. But maybe that is ok (or even good).

Assume I have a build that depends on "jackson-core" and "LibraryX"; whrere "LibraryX" transitively depends on "jackson-jaxrs-json-provider".

Now assume "jackson-core:2.11" has just been published and I update my build to use it. With the BOM dependency, I would then get "jackson-bom:2.11". By that, also the transitive dependency will be upgraded to "jackson-jaxrs-json-provider:2.11" (because there is an entry for that in the BOM). If "jackson-jaxrs-json-provider:2.11" has not been published yet, Gradle will fail trying to find the component. However, this might actually be good: you should not upgrade to "2.11" until all Jackson components you use are available in that version.

(If you still want to do take the risk of using potentially incompatible versions, you can add a constraint to your own build to "override" what you get from the BOM)

[Tatu Saloranta]

Right. There is a problem wrt lack of atomicity, inherent unless BOM
itself was used as the "commit", in a way. But given other challenges
I think it is a workable approach.

> Assume I have a build that depends on "jackson-core" and "LibraryX"; whrere "LibraryX" transitively depends on "jackson-jaxrs-json-provider".
>
> Now assume "jackson-core:2.11" has just been published and I update my build to use it. With the BOM dependency, I would then get "jackson-bom:2.11". By that, also the transitive dependency will be upgraded to "jackson-jaxrs-json-provider:2.11" (because there is an entry for that in the BOM). If "jackson-jaxrs-json-provider:2.11" has not been published yet, Gradle will fail trying to find the component. However, this might actually be good: you should not upgrade to "2.11" until all Jackson components you use are available in that version.
> (If you still want to do take the risk of using potentially incompatible versions, you can add a constraint to your own build to "override" what you get from the BOM)

Yes.

-+ Tatu +-

>
> --
> You received this message because you are subscribed to the Google Groups "jackson-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to jackson-user...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/jackson-user/6a9e9fb5-9653-4589-bf50-f9b901887b36%40googlegroups.com.

[Jendrik Johannes]

Hi,

After discussing this a bit more with my team, I had a look at providing a Maven plugin for this and here it is:

https://github.com/jjohannes/gradle-module-metadata-maven-plugin

I am finishing that up and will publish a first version soon.

What would be the best way to proceed then. Should I try it with all the Jackson projects that use the BOM and if it works as expected open PRs on all projects?

> To unsubscribe from this group and stop receiving emails from it, send an email to jackso...@googlegroups.com.

[Jendrik Johannes]

I am about to publish a first version of the plugin. (Have some connection problems right now, but should be done soon.)

In theory you could add it to 'jackson-base' as it should be the same for all components:

https://github.com/FasterXML/jackson-bom/blob/master/base/pom.xml

This would be the configuration to add:

------------------------------------------

      <plugin>

        <groupId>de.jjohannes</groupId>

        <artifactId>gradle-module-metadata-maven-plugin</artifactId>

        <version>0.1.0-SNAPSHOT</version>

        <executions>

          <execution>

            <goals>

              <goal>gmm</goal>

            </goals>

          </execution>

        </executions>

        <configuration>

          <platformDependencies>

            <dependency>

              <groupId>com.fasterxml.jackson</groupId>

              <artifactId>jackson-bom</artifactId>

              <version>${project.version}</version>

            </dependency>

          </platformDependencies>

        </configuration>

      </plugin>

------------------------------------------

Is there some good way to test this locally? Like a script to clone all (or some) repositories and install them? How do you usually test changes to 'jackson-base'?

[Tatu Saloranta]

On Wed, Feb 26, 2020 at 1:43 PM Jendrik Johannes <jen...@gradle.com> wrote:

I am about to publish a first version of the plugin. (Have some connection problems right now, but should be done soon.)

In theory you could add it to 'jackson-base' as it should be the same for all components:

https://github.com/FasterXML/jackson-bom/blob/master/base/pom.xml

I dropped the ball here, but getting back... would you mind filing an issue for `jackson-databind`:

https://github.com/FasterXML/jackson-databind/issues

just cut-n-pasting this setting? I think I could get this added for 2.12 (just released 2.11.0 last weekend, so can focus on next minor version), starting testing now.

-+ Tatu +-

 

To unsubscribe from this group and stop receiving emails from it, send an email to jackson-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jackson-user/3992e8f9-0e0b-4cf3-b3a6-2ed7dde22882%40googlegroups.com.

[Jendrik Johannes]

Hi Tatu,

Thanks for getting back. I also wanted to follow up creating an issue/PRs, but did not get around to do it. It's done now: https://github.com/FasterXML/jackson-databind/issues/2726

Let me know if there are further questions or if I can help with anything. I am certainly happy to test any changes with Gradle. It's probably the easiest to further discuss on the issue.

Regards,

Jendrik

To view this discussion on the web visit https://groups.google.com/d/msgid/jackson-user/3992e8f9-0e0b-4cf3-b3a6-2ed7dde22882%40googlegroups.com.

[Tatu Saloranta]

Thank you! I'll have a look, thank you for helping here!

-+ Tatu +-

> To unsubscribe from this group and stop receiving emails from it, send an email to jackson-user...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/jackson-user/89124e92-3324-4702-b907-cc5eb97bbf31%40googlegroups.com.