Maven central artifacts and maven support in bndtools

[Christian Schneider]

I would like to move from some artificial test projects I did with bndtools to something more realistic.

To do that I would like to leverage some Aries and pax artifacts to provide DataSource and JPA/JTa support.

As these artifacts live in maven central I wonder what the best way is to include them?

Can I use a pom based approach for build time? I read in older posts that this might work.

How are build time dependencies resolved then?

If I stay with pure bndtools how do I access the maven artifacts?

I looked for https://jpm4j.org/ but it does not even load in my browser because of ssl issues.

As always any help or pointers to examples are highly appreciated.

Christian

--

--
Christian Schneider
http://www.liquid-reality.de

Open Source Architect
http://www.talend.com

[Pierre De Rop]

Hello Christian;

I'm not sure if this is the best way to get access from Maven Central from a pure bndtools project, but here is how I'm doing this:

I'm using the aQute.bnd.deployer.repository.aether.AetherRepository like this:

1) first, edit your plugin path and configure the following:

bnd-release    = https://bndtools.ci.cloudbees.com/job/bnd.master/lastSuccessfulBuild/artifact/dist/bundles

-pluginpath: \
    ${plugin-dir}/biz.aQute.repository/biz.aQute.repository.jar,\
    ${build}/cache/biz.aQute.repository.aether-latest.jar;url=${bnd-release}/biz.aQute.repository.aether/biz.aQute.repository.aether-latest.jar

2) next, add the AetherRepository to your ext/repositories.bnd:

-plugin:\
    ...,\
    aQute.bnd.deployer.repository.aether.AetherRepository; name=Maven Central; url=http://repo1.maven.org/maven2/

3) then from your bnd project which needs build dependencies from maven central, you can then define maven dependencies using the following format:

-buildpath:  \
    <groupid>:artifactId;version=<version>

For instance, the following defines two dependencies to maven central artifacts:

-buildpath:  \
    org.osgi:org.osgi.core;version=6.0.0,\
    org.osgi:osgi.cmpn;version=6.0.0,\
    ...

You can refer to [1] for a concrete example, where the repositories.bnd ([2]) includes the AetherRepository, and the subproject from [3] defines some dependencies from maven central.

Now, if you need to use an http proxy in order to go outside your intranet, you can configure your own local nexus that can proxy to maven central and configure the AetherRepository with it.
I don't know currently if there is an easier way to do that and I asked this in this mailing list (see post "Is is possible to use an http proxy with the AetherRepository ?" from [4]).

 

Hope this helps;

cheers;

/Pierre

[1] http://svn.apache.org/viewvc/felix/sandbox/pderop/dependencymanager.ds/
[2] http://svn.apache.org/viewvc/felix/sandbox/pderop/dependencymanager.ds/cnf/ext/repositories.bnd
[3] http://svn.apache.org/viewvc/felix/sandbox/pderop/dependencymanager.ds/org.apache.felix.dependencymanager.ds/bnd.bnd
[4] https://groups.google.com/forum/#!topic/bndtools-users/Ls0kAEBIKpw

--
You received this message because you are subscribed to the Google Groups "bndtools-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bndtools-user...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Christian Schneider]

Hi Pierre,

thanks for the descriptions. That should work for the build path. Will try it soon.

How does the launcher work with this though? As far as I understood bndtools needs the OBR like meta data to determine the runtime artifacts and the aerther repository does not provide them.

Christian

[Pierre De Rop]

good question. I don't know actually; Indeed, the bndtools OBR probably can't resolve using the aether repository ?

I just checked the bndtools tutorial from [1], and it says that if an index is not available, then the repository can't participate in run requirement resolution ...

cheers;

/Pierre

[1] http://bndtools.org/repositories.html#aether-maven-repositories

[Neil Bartlett]

That's right, resolving requires an index. Of course you can always explicitly list your bundles in -runbundles, and these can be picked up from anywhere including an AetherRepository.

Incidentally we are working very hard to improve Maven support in Bndtools 3.1, it will be the major theme of the next release. If any hardcore Maven users are interested in helping out, that would be awesome. Not just to help us, but also to make sure that the release does what you need it to do.

Neil

[Christian Schneider]

I am willing to help as well as I can.

I think for the runtime resolution we need at least some assistence by
bndtools. So it should be possible to search in the maven repository so
the maven coordinates do not have to be typed blindly. It would also be
nice to be able to add groups of bundles in one step. Eclipse as well as
karaf features could help there. I guess bdntools definitely does not
yet support karaf features but does it support Eclipse features and p2
repos?

I think it is unrealistic for maven central to have one complete OSGi
metadata index. It would be too large and probably even contain a lot of
problematic or contradicting information. Would it be possible to
instead create indexes for subsets of bundles. Like Aries could create
one index for each subproject per version. This index could be generated
from a list of bundles that will also be used during the testing. Does
that make sense?

Christian

On 13.09.2015 23:28, Neil Bartlett wrote:
> That's right, resolving requires an index. Of course you can always
> explicitly list your bundles in -runbundles, and these can be picked
> up from anywhere including an AetherRepository.
>
> Incidentally we are working very hard to improve Maven support in
> Bndtools 3.1, it will be the major theme of the next release. If any
> hardcore Maven users are interested in helping out, that would be
> awesome. Not just to help us, but also to make sure that the release
> does what you need it to do.
>
> Neil
>
>

[Peter Kriens]

You do not need to use the -pluginpath anymore for the biz.aQute.repository repository. We now include it in bndtools, bnd, and gradle.

And some very good news, we’re working on the Aether repository now to include it as well and fully support releasing to local, remote, snapshot and normal repos taking all maven settings into account. At the bnd hackathon this week we set the goal to fully support maven. We want to provide:

• Maven first — The wonderful bndtools ‘bndrun’ resolving/debugging/exporting experience. We made the bndrun editor standalone and provide a maven plugin to index a set of maven artifacts defined in a pom. We will also provide an export maven plugin to convert a bndrun to an executable JAR or other executable format supported by bnd.
• Bndtools first — Fully support releasing/consuming to/from maven snapshot/local/remote/normal repositories, taking all maven settings into account. This will make a variation on the Aether repository a first class citizen in bnd(tools).

Kind regards,

Peter Kriens

[Christian Schneider]

That is really good news. Let me know if I can help with testing or reviewing the concepts.

Christian

[Neil Bartlett]

Responses inline…

On 14 Sep 2015, at 08:15, Christian Schneider <ch...@die-schneider.net> wrote:

I am willing to help as well as I can.

I think for the runtime resolution we need at least some assistence by bndtools.

Yes. One of the new features I am working on now in Bndtools 3.1 is a standalone mode for bndrun files. The problem with the current version is that bndrun files must be connected to a bnd workspace with cnf folder etc, which you are unlikely to have in a Maven build. In the new standalone mode, you just need to have a list of repository indexes in the bndrun file. Bndtools will support resolving and running from this file, and it works consistently whether or not you have a cnf. For example:

-standalone: \

http://example.org/myindex.xml.gz, \

${user.home}/.m2/repository/repository.xml

So it should be possible to search in the maven repository so the maven coordinates do not have to be typed blindly. It would also be nice to be able to add groups of bundles in one step. Eclipse as well as karaf features could help there. I guess bdntools definitely does not yet support karaf features but does it support Eclipse features and p2 repos?

Bndtools does not support either Karaf features or Eclipse features. I’m not sure what we would use these for.

We may in the future support p2 repositories as an alternative index type, and this might help interop with PDE. I don’t think this is directly related to Maven however. Personally I don’t plan to spend any time on p2 integration unless a customer sponsors the development of it.

I think it is unrealistic for maven central to have one complete OSGi metadata index. It would be too large and probably even contain a lot of problematic or contradicting information.

Yes absolutely. Even your ~/.m2/repository directory is very uncontrolled and likely to produce unpredictable resolution results.

Our plan at the moment — and Tim Ward has been working on this — is to deliver a separate Maven plugin called the bnd-indexer-maven-plugin. This works a bit like the assembly plugin, except that instead of spitting out an EAR or WAR file it produces an OSGi index of its (transitive) dependencies. So you would create an index module in your top-level project that lists dependencies on your own modules, and it hopefully indexes everything required including third-party dependencies from Maven Central or your Nexus/Artifactory repo.

Unfortunately the dependencies in third party POMs are often poor quality, so the transitive dependency set will likely miss some things. In these cases you would have to add them as explicit dependencies to your index module. On the other hand, it doesn’t really matter if you overshoot and index too many dependencies, because the Bndtools resolver will still resolve only the minimal set for your top-level dependencies.

Would it be possible to instead create indexes for subsets of bundles. Like Aries could create one index for each subproject per version. This index could be generated from a list of bundles that will also be used during the testing. Does  that make sense?

Yes. The missing pieces in this tool chain at the moment are the ability to take a bndrun file and: (1) integration test it, and (2) assemble a deployable & executable artifact such as an uber-jar.

Christian

On 13.09.2015 23:28, Neil Bartlett wrote:

That's right, resolving requires an index. Of course you can always explicitly list your bundles in -runbundles, and these can be picked up from anywhere including an AetherRepository.

Incidentally we are working very hard to improve Maven support in Bndtools 3.1, it will be the major theme of the next release. If any hardcore Maven users are interested in helping out, that would be awesome. Not just to help us, but also to make sure that the release does what you need it to do.

Neil

--
Christian Schneider
http://www.liquid-reality.de

Open Source Architect
http://www.talend.com

[Christian Schneider]

I just tried to remove the pluginPath but it does not seem to work

Error:
Cannot load the plugin aQute.bnd.deployer.repository.aether.AetherRepository

This is my config with the pluginpath commented out.

#bnd-release    = https://bndtools.ci.cloudbees.com/job/bnd.master/lastSuccessfulBuild/artifact/dist/bundles
#-pluginpath: \
#    ${plugin-dir}/biz.aQute.repository/biz.aQute.repository.jar,\
#    ${build}/cache/biz.aQute.repository.aether-latest.jar;url=${bnd-release}/biz.aQute.repository.aether/biz.aQute.repository.aether-latest.jar
   
-plugin: \
    aQute.bnd.deployer.repository.LocalIndexedRepo;name=Release;local=${workspace}/cnf/releaserepo;pretty=true,\
    aQute.bnd.deployer.repository.LocalIndexedRepo;name=Local;local=${workspace}/cnf/localrepo;pretty=true,\
    aQute.bnd.deployer.repository.FixedIndexedRepo;name=Bndtools Hub;locations=https://raw.githubusercontent.com/bndtools/bundle-hub/master/index.xml.gz,\
    aQute.lib.deployer.FileRepo;name=Build;location=${workspace}/cnf/buildrepo;latest=false,\

    aQute.bnd.deployer.repository.aether.AetherRepository; name=Maven Central; url=http://repo1.maven.org/maven2/

On 14.09.2015 11:33, Peter Kriens wrote:

[Christian Schneider]

On 14.09.2015 11:37, Neil Bartlett wrote:
>
> Bndtools does not support either Karaf features or Eclipse features.
> I’m not sure what we would use these for.
>
> We may in the future support p2 repositories as an alternative index
> type, and this might help interop with PDE. I don’t think this is
> directly related to Maven however. Personally I don’t plan to spend
> any time on p2 integration unless a customer sponsors the development
> of it.
>

The features allow to add functionality in a coarse grained way. In
karaf this is crucial as we do not have an index and a real resolver. So
without features it would be just too much work to add all needed
bundles. Karaf 4 now works with a resolver internally so that might
shift a bit in the future.

Still I think features are necessary sometimes. For example if you have
a very loosely coupled architecture then the resolver can not always
help you to choose a good set of bundles. With an index the karaf
features could be changed. So instead of containing all detailed bundles
they would just contain requirements like the -runrequires in bndtools.
Bndtools could then allow to add features as requirements so people do
not have to add all of that themselves.

So for example if your user bundle just depends on the persistence api
and the transaction api then the resolver can not help much to install
Aries JPA and for example hibernate. What is your strategy in bndtools
to solve this without features?

>> I think it is unrealistic for maven central to have one complete OSGi
>> metadata index. It would be too large and probably even contain a lot
>> of problematic or contradicting information.
>
> Yes absolutely. Even your ~/.m2/repository directory is very
> uncontrolled and likely to produce unpredictable resolution results.
>
> Our plan at the moment — and Tim Ward has been working on this — is to
> deliver a separate Maven plugin called the bnd-indexer-maven-plugin.
> This works a bit like the assembly plugin, except that instead of
> spitting out an EAR or WAR file it produces an OSGi index of its
> (transitive) dependencies. So you would create an index module in your
> top-level project that lists dependencies on your own modules, and it
> hopefully indexes everything required including third-party
> dependencies from Maven Central or your Nexus/Artifactory repo.
>
> Unfortunately the dependencies in third party POMs are often poor
> quality, so the transitive dependency set will likely miss some
> things. In these cases you would have to add them as explicit
> dependencies to your index module. On the other hand, it doesn’t
> really matter if you overshoot and index too many dependencies,
> because the Bndtools resolver will still resolve only the minimal set
> for your top-level dependencies.

I really like the idea to be able to create a custom index from a maven
pom. I also had that in mind.
Additionally we should push these indexes to the maven repo during the
build. So for example Aries can create indexes for their subprojects
that make it then really easy for people to use them in bndtools.

>> Would it be possible to instead create indexes for subsets of
>> bundles. Like Aries could create one index for each subproject per
>> version. This index could be generated from a list of bundles that
>> will also be used during the testing. Does that make sense?
>
> Yes. The missing pieces in this tool chain at the moment are the
> ability to take a bndrun file and: (1) integration test it, and (2)
> assemble a deployable & executable artifact such as an uber-jar.

If the maven plugin you mentioned creates the index wouldnt bndrun then
be able to do the rest? As far as I know integration tests and the
assembly already work for classic bndtools indexes.

Christian

[Pierre De Rop]

Christian,

I think Peter was talking about just the ${plugin-dir}/biz.aQute.repository/biz.aQute.repository.jar, which is not required anymore on the plugin path. See the changelogs in [1].

So, simply using the AetherRepository jar in the buildpath should be enough; like this:

bnd-release    = https://bndtools.ci.cloudbees.com/job/bnd.master/lastSuccessfulBuild/artifact/dist/bundles

-pluginpath: \

    ${build}/cache/biz.aQute.repository.aether-latest.jar;url=${bnd-release}/biz.aQute.repository.aether/biz.aQute.repository.aether-latest.jar

cheers;

/Pierre

[1] https://github.com/bndtools/bndtools/wiki/Changes-in-3.0.0