Improving the user experience for maven standalone mode

[Christian Schneider]

Recently at the OSGi event in Darmstadt I talked to BJ and some others about how the pure maven builds can move nearer to the user experience of a native bndtools build.

So lets start with the current status.

I currently use the bnd-indexer-plugin to create a local index with the setting localUrls REQUIRED. This creates a good quality index even for maven SNAPSHOTS. The downside
is that the index can not be published as it has system specific local URLs but that is no issue for me as I only create the index local to the bndrun file that uses it.

So with that approach I can create a pure maven build that looks a lot like normal OSGi builds at apache. I can then have a run and "assembly" project that uses a bndrun file to define my application.
I can resolve, run and debug the application. Using a maven plugin I can also resolve and export the application as a runnable jar.

Now to the limitations:
- If the meta data of a bundle changes I might have to recreate the index from maven. ( Quite seldom in practice).
- I have to add the source of my projects manually to the debug session. The source of external projects sometimes does not show at all.
- When I change any code I have to do mvn install on the individual project and restart my debug session. (Compared to just saving the source file in native bndtools build).

So it kind of works already but is a lot less convenient than the native bndtools style. On the positive side it feels very maven like. So any maven user should get along very well with this kind of build.

Of course the goal should be to bring the user experience to the same level as native bndtools. So what can we do?

Automatically (re)create the index from the eclipse build

If the bnd-indexer-plugin takes part in the incremental build then it might be able to:
1. create the index initially when the maven project is imported into eclipse (to avoid having to run mvn install for the index initially)
2. react to changes in dependencies and automatically rebuild the index

Automatically attaching source

In maven the source code of bundles is not inside the binary bundle but in a separate bundle. M2e automatically downloads sources for normal maven runs. It would be great if the bndtools runner could hook into the same mechanism to load and attach the source of dependencies. Eventually this can be done in the project creating the index as this project has all sources as maven or project dependencies. So the m2e mechanisms might already do most of the work there. Maybe it is good enough to automatically attach the source of the index poject to the run. Does that make sense?

Treating workspace projects specially

One interesting thing in m2e is that it treats dependent projects differently if they are opened in the workspace or not. We should do the same thing for bndrun. If the project is not open in the workspace the bundle should be used from the local maven repo. The bnd indexer plugin makes sure all external dependencies are resolved and loaded into the local repo automatically. So this makes all external projects available in an efficient way. As snapshots are always deployed into the same file in the local repo this also allows to directly use new snapshots if they are updated from a maven build.
If the project is opened in the workspace we should use the exploded bundle in the target/classes dir. This allows direct access to code built by the eclipse build and avoids creating jar files after each change.
So some question here are:
- Can the bndtools runner/debugger work with such exploded bundles?
- Can I just give it the path?
I think we would need to adjust the path in the indexer plugin .. Not yet sure how to do this. I would be grateful for any ideas.

Doing full incremental build and updating the bundle in the debugger/run

When saving any file in a project m2e will trigger all maven plugins registered for incremental builds. These then have to make sure they recreate any changed resources.
I found that the bnd-maven-plugin sometimes seems to have issues there.
Sometimes I get a Manifest without any OSGi meta data. I think this is a bug in the plugin. https://github.com/bndtools/bnd/issues/1551
It also seems the Manifest is never placed in target/classes https://github.com/bndtools/bnd/issues/1552

Alerting the bndtools runner when a bundle changes to trigger an update of the bundle in OSGi:
This would be beneficial in two cases.
1. A local maven build is done and changes some SNAPSHOTS in the local maven repo. The runner should detect these changes and update the bundles.
2. A file is saved in eclipse and the incremental build produces new code. The runner should then updated the exploded bundle

I would be happy about any feedback and with some guidance I can probably help to code some of these improvements.

Christian

-- 
Christian Schneider
http://www.liquid-reality.de

Open Source Architect
http://www.talend.com

[BJ Hargrave]

I am unsure of the development environment you are talking about. Is it M2E for Eclipse use? Since it is pure maven build, Bndtools does not apply, so I would assume any Eclipse related changes would be to M2E. Or are you proposing Bndtools be "enhanced" to be used, in alternate to M2E, for pure maven bundle development? Or is there some hybrid model you are proposing?

--
You received this message because you are subscribed to the Google Groups "bndtools-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bndtools-user...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

--
BJ

[David Daniel]

From the changes that Christian is proposing the ones that I would have the most interest in is mangling of the index created by the bundle and an eclipse workspace.  Examples from above would be having the index created by the bnd-index-plugin point at an exploded jar.  Takari has done some work with osgi, eclipse, maven and exploded jar's here  https://github.com/takari/nexus-plugin-bundle-maven-plugin  but I am not familiar enough with Maven or bnd to know if it is applicable.  The user use case for me is as follows.  When working with a bnd.bnd file when I make a change the bundle is automatically redeployed and I can keep debugging.  With a bndrun file because the index points at a jar and not the exploded code that eclipse uses in the workspace I have to do a mvn build or install to debug my code.  M2e will autmatically rebuild my file in exploded form when I change a file that effects my build and it would be nice if there were a way to tell the bnd-indexer-plugin to point at an exploded version in my workspace rather than a built jar.

Another thing I was wondering about is if this scenario that I would like to do possible with 1 build in a ci environment. 

build bundles that bnd-indexer depends on
Run bnd-indexer-plugin
have a bndrun file that uses the output of the bnd-indexer-plugin as its repo
Run bndtools to generate a megajar from the bndrun file

I am pretty sure this is possible but I have not had time to implement it yet. Like Christian I have had trouble with the output index file not appearing in the target folder so when other steps run the dependency will be there.  I think it is because my lack of knowledge and user error though rather than a bug.  I don't think I understand the lifecycle well enough.

[Christian Schneider]

Yes. I am talking about integration with M2E. I think we do not need to
change M2E. The incremental build API should provide all the hooks we
need. Actually the bnd-maven-plugin seems to
create everything in target/classes we need already.

So like David wrote we just need the index to point at target/classes
for the projects that are part of the eclipse workspace. Then at least
starting the run/debug with exploded bundles should be easy.
The only thing remaining there would be to give the runner a
notification that a project has changed and the bundle must be updated
after the incremental build is finished.

I think I should be able to change the bnd-indexer-plugin to take part
in the incremental build and detect which bundles are part of the
workspace. I will experiment a bit with this and report my findings.

Could you maybe already describe how the bndtools runner can be notified
to reload bundles? I am not yet sure how to hook into the build to know
it is finished but plain bndtools definitely already has this
functionality. So I can imagine that we simply need a very simple
bndtools builder for that case that lets M2E + bnd-maven-plugin do the
changes in target/classes and then just notifies the runner of the
changed bundle.

Christian

On 15.07.2016 15:11, BJ Hargrave wrote:
> I am unsure of the development environment you are talking about. Is
> it M2E for Eclipse use? Since it is pure maven build, Bndtools does
> not apply, so I would assume any Eclipse related changes would be to
> M2E. Or are you proposing Bndtools be "enhanced" to be used, in
> alternate to M2E, for pure maven bundle development? Or is there some
> hybrid model you are proposing?
>

[Christian Schneider]

If I understand you correctly then you simply want a maven build that
creates the individual bundles, the index and the runable jar with all
the bundles embedded.
This already works. My osgi-chat project does exactly this:
https://github.com/cschneider/osgi-chat/

This creates the index:
https://github.com/cschneider/osgi-chat/blob/master/packaging/index/pom.xml

This is my bndrun
https://github.com/cschneider/osgi-chat/blob/master/packaging/all/chat-all.bndrun

and this creates the runable jar
https://github.com/cschneider/osgi-chat/blob/master/packaging/all/pom.xml

I use a custom maven plugin for this case but I already have done a pull
request to include this plugin in bnd. As Peter approved the plugin I
would expect it to be part of bndtools 3.3.
In the mean time you can use my plugin to try it.
https://github.com/cschneider/osgi-maven-plugins/tree/master/bndrun-maven-plugin

Christian

On 15.07.2016 15:41, David Daniel wrote:
>
> Another thing I was wondering about is if this scenario that I would
> like to do possible with 1 build in a ci environment.
>
> build bundles that bnd-indexer depends on
> Run bnd-indexer-plugin
> have a bndrun file that uses the output of the bnd-indexer-plugin as
> its repo
> Run bndtools to generate a megajar from the bndrun file
>
> I am pretty sure this is possible but I have not had time to implement
> it yet. Like Christian I have had trouble with the output index file
> not appearing in the target folder so when other steps run the
> dependency will be there. I think it is because my lack of knowledge
> and user error though rather than a bug. I don't think I understand
> the lifecycle well enough.

[David Daniel]

Thank you very much.  You saved me a lot of time.

David

--

You received this message because you are subscribed to a topic in the Google Groups "bndtools-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/bndtools-users/_xsTCxn-lzw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to bndtools-user...@googlegroups.com.

[Christian Schneider]

I just quickly tested using an exploded bundle

In the index.xml I changed
<attribute name="url"
value="file:/home/cschneider/checkout/osgi-chat/command/target/net.lr.demo.chat.command-1.0.0-SNAPSHOT.jar"/>
into
<attribute name="url"
value="file:/home/cschneider/checkout/osgi-chat/command/target/classes"/>

I started my bndrun. It still works fine. So the exploded bundles should
be no issue.
Then I did a change in a java file and just saved it without triggering
the maven build. The incremental build still made sure that the classes
and Manifest are updated.
As the bundle is not updated automatically I did "update 3" on the gogo
shell.
After that my changed code was running.

So we really just need to make sure the index points to target/classes
which is easy and hook into the build to signal the runner that the
bundle needs to be reloaded.
I will prepare the changes for the bnd-indexer-plugin.

Christian

[David Daniel]

It would be helpful if it was based off a maven property.  That way on the build server even if the bnd-indexer-plugin is pointing at a project in the workspace it will use the built jar but if maven has a live deploy param then it points at the exploded jar.

[Peter Kriens]

Please note that I expect that the new BndPomRepository plugin in bnd will largely replace the functionality of the index plugin.

Kind regards,

Peter Kriens

[Neil Bartlett]

BJ, you imply that Bndtools can not provide value in an m2e environment, which pits them as competitors. I think that's unhelpful. With m2e we are no longer in charge of the build, but we can still run/debug bndrun files, resolve, inspect bundles, provide assisted editing of bnd files, etc.

Neil

[BJ Hargrave]

My point is that someone is in charge of the building. Bndtools is in change for Bnd Workspace model builds and M2E is in charge for maven builds. Yes, parts of Bndtools could be used in non-Bnd Workspace model builds. But when Christian is talking about incremental builders, that is something different. You really cannot have both Bndtools incremental builders and M2E incremental builder both trying to be in charge of building a project.

Bndtools is nice in that it build the actual bundle in the incremental builders. As I understand M2E, it does not build the jar artifact incrementally which may be what Christian is missing. During incremental building, M2E invokes the bnd-maven-plugin which processes target/classes properly but does not build the jar which would be needed by other bnd-*-maven-plugins like indexing and baselining.

[Christian Schneider]

I was not talking about the bndtools incremental builders but on the maven one. The bnd-maven-plugin already takes part in the M2E incremental build. So after each such incremental build target/classes should be in a good state. So I think we just need a very simple hook for bndtools that is invoked when the incremental build has finished. Bndtools then needs to notify the launcher to update the bundle. That should be quite simple but I do not know enough about the eclipse build internals to do that.

It would be great if indexing could be done during incremental builds too but I think it is would not be a big problem if the user would need to launch maven in this case. Maybe Peters pom based index can cover this. Baselining should be possible as part of the m2e incremental build. The maven api for it even allows to mark errors in files.

So I think we are not really far from providing a really good user experience in maven builds.

Christian

[BJ Hargrave]

M2E already has the incremental builder and it calls maven plugins. So rather than inventing a special way to call bndtools when the m2e incremental builder is done, the proper way to handle this would be to write a maven plugin which integrates with the M2E lifecycle to notify the launcher. This maven plugin would be a no-op in offline builds.

[Christian Schneider]

Do wee need a new plugin for this or can the bnd-maven-plugin do this? If you can tell me what it needs to do I can look into it.

Christian

[BJ Hargrave]

You may be able to do it with the same plugin, but I think you will need a different mojo class to hook into a different part of the build lifecycle [1]. bnd-maven-plugin's current mojo is part of the process-classes phase [2]. I think you would need to hook into the phase after the package phase when the jar is made.

[1] https://maven.apache.org/guides/introduction/introduction-to-the-lifecycle.html#Lifecycle_Reference
[2] https://github.com/bndtools/bnd/blob/master/maven/bnd-maven-plugin/src/main/java/aQute/bnd/maven/plugin/BndMavenPlugin.java#L61

[Christian Schneider]

Yes.. the phase is a problem for reusing the bundle plugin. It will probably make sense to experiment with a separate plugin first anyway. Can you give me some hints how I can trigger the reloading of bundles in the launcher?

Christian

[David Daniel]

I think I understand better now how things are working and what I want which of course is the world.  What I currently do is as follows and the reasons behind it.  I maintain two build systems Maven and Bnd.  Before commits I give my projects a maven nature and build it with that. 

• This makes sure my pom has the correct dependencies.
  
• It verifies that the build will work in jenkins which is what I use for deployment and ci.
  
• It verifies my integraion tests with pax-exam and unit tests with surefire.
• It does my karaf build that builds my custom karaf distribution
  

Development time I give the project a bndtools nature.

• This finds errors in my bnd.bnd file.
• It allows me to use the bnd.bnd file and bndrun file for debugging.
• It verifies that if I want to produce a megajar it will work.

I understand that I can't have both natures because they can conflict.  It is not the case with all project natures as I can have a project that combines maven and scala or javascript but both maven and bnd are fighting for what belongs in the classpath.  I know I can do the second two bullet points under the bndnature in maven now.  From Christians note above I can build a single jar.  Using the debugging tips for debugging karaf in bndtools on the enroute site I can create a custom lifecycle in maven and run pax-exams karaf container.  A combination of https://books.sonatype.com/mvnref-book/reference/writing-plugins-sect-plugins-lifecycle.html and https://github.com/ops4j/org.ops4j.pax.exam2/tree/master/containers  I would prefer to let maven handle the build for the following reasons. 

• It is more widely supported.
• It works in more than just standard eclipse.  It would be nice to have a build that can work in something like eclipse che or intellij as well.
• There is more maven knowledge at our company then osgi (Just me banging the drum :( )

That being said I love the bndtools experience.  I want the best of both worlds and would love to get all the error messages, warnings, repository views and other great things in bndtools even with the maven nature turned on.  I think in order to do this I would have to have a way to create the bnd workspace from maven.  I think I can do this using a custom plugin that parses my parent pom and creates the appropriate x.maven files and updates the build.bnd file.  I can hide the warning that shows when there is both a maven nature and bnd nature.  The problem I am not sure how to fix is that both m2e will be building and saving jars as well as bnd.  I currently have them setup to build and save to the same file but this can cause conflicts.  Is there a way to have the bnd nature on a project and have bnd build and check for errors but not persist the jar.  Or is there a way to get the repository view and bnd build tips without having a bnd project nature on the individual projects.

Thanks for any help,

  David