Apache Karaf and bndtools

[Christian Schneider]

I read about the problems of combining karaf and bndtools and thought a
bit about possible solutions.

So if I understood correctly then one issue is that karaf provides quite
a few bundles by default that might then be duplicated by
what the resolver tries to install.

I hope the -distro option can help with that. One other thing we should
try is to use a very slim karaf. There is already a karaf minimal distro
but we can make it even smaller. Using a karaf custom distro we can
strip down karaf to just the framework and the starter of course. Then
the bndtools resolver could do the full provisioning. If there is
interest in it we can
provide such a bare bone distro by default in the karaf project so it is
easy to use by bndtools users.

Another step then would be to provide the karaf contents as an index or
pom so bndtools can provision the needed parts of karaf.
So this should then deliver a good list of runbundles to install.

As far as I know this is currently the only doc of how to combine karaf
and bndtools:
http://enroute.osgi.org/appnotes/bndtools-and-karaf.html

This only provides a way to debug karaf though and it also needs a few
manual steps.
So I wonder how a complete solution would look like.

For debug I think we need to automate starting karaf when pressing debug
and installing the remote agent.
For export we need to do the following:
- Extract karaf bare bone
- Provision the runbundles into the system dir
- Reference the runbundles in startup.properties so they are actually
installed and started
- Add other parts of the bndrun into the respective karaf configs like
system.packages.extra and similar and add the runpath bundles into the
lib dir of karaf.
There will be quite a few things there I guess

Can anyone point me to some documentation how this special behaviour
could be plugged in?

Christian

--
Christian Schneider
http://www.liquid-reality.de

Open Source Architect
http://www.talend.com

[Sascha Homeier]

Hi Christian,

this sounds very interesting.

Currently I am using a very messy approach by looking at what bundles are running in Karaf 4.0.4 and using the same versions in my runbundles.

This of course might require a manual update of my runbundles when I update to another Karaf distribution.

For provisioning my bundles into Karaf I use a Karaf-Feature XML which is installed into a pre-configured Karaf 4.0.4 distribution and then packaged as RPM.

To not always need to adjust the Karaf-Distribution I use some Gradle-Scripts to adjust/add Strings in Karaf configuration (for example in system.properties etc).

Anyway a lot of manual steps still need to be done to achieve "fidelity" between my bndtools run environment and Karaf.

As with bndtools 3.2/3.3 the Maven support was enhanced I am now trying to improve the packaging/provisioning process by using the karaf maven plugin for feature xml and customizing the Karaf Distribution.

As far as I have understood I now can access my bndtools projects as dependencies from a Maven project (which then builds the Karaf distribution).

Anyway I think things like "system.packages.extra" might then still need to be managed twice: In the pom.xml for the Karaf Dristribution and the bnd rundescriptor.

So an export like you described it would be very nice. But I am not familiar how this could be plugged in to bndtools (maybe just by using the Eclipse Extension Points).

[Thomas Driessen]

Hi,

has there been any progress regarding this subject? 

Kind regards,

Thomas

[Raymond Auge]

I don't think the problem is any different than with any pre-build container.

Peter and I just documented the process on the enroute site w.r.t. resolving under the heading Distro [1].

- Ray

[1] http://enroute.osgi.org/appnotes/resolving.html#distro

--
You received this message because you are subscribed to the Google Groups "bndtools-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bndtools-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Raymond Augé (@rotty3000)

Senior Software Architect Liferay, Inc. (@Liferay)

Board Member & EEG Co-Chair, OSGi Alliance (@OSGiAlliance)

[Thomas Driessen]

Thank you Ray. I will have a look at this.

To unsubscribe from this group and stop receiving emails from it, send an email to bndtools-user...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Milen Dyankov]

I don't think the problem is any different than with any pre-build container.

It is different in 2 ways:

 - Karaf has features - which is what most people use to install bundles in Karaf
 - Karaf allows you to build a custom distributions  - which is AFAIK the way most people use Karaf

If you need any of those for whatever reason, the process Ray and Peter describe would only get you half way there. 

If you only need to deploy bundles into running Karaf container than yes, it's no different than any other container. 

 

Best,

Milen

To unsubscribe from this group and stop receiving emails from it, send an email to bndtools-users+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

http://about.me/milen

[Raymond Auge]

On Wed, Dec 6, 2017 at 6:16 PM, Milen Dyankov <milend...@gmail.com> wrote:

I don't think the problem is any different than with any pre-build container.

It is different in 2 ways:

 - Karaf has features - which is what most people use to install bundles in Karaf
 - Karaf allows you to build a custom distributions  - which is AFAIK the way most people use Karaf

If you need any of those for whatever reason, the process Ray and Peter describe would only get you half way there. 

Why? Just setup all the features you want, then run the distro operation... then use that to resolve!

The from the resulting bundles... make a feature!

I don't see the problem.

- Ray

[Milen Dyankov]

 

The from the resulting bundles... make a feature!

 ... and than add that feature to the build + configuration to generate custom Karaf distribution. 

That is exactly why I said "will get you half way there" :) 

[Raymond Auge]

On Wed, Dec 6, 2017 at 6:26 PM, Milen Dyankov <milend...@gmail.com> wrote:

 

The from the resulting bundles... make a feature!

 ... and than add that feature to the build + configuration to generate custom Karaf distribution. 

That is exactly why I said "will get you half way there" :) 

My point is that Karaf is not special here... Any container likely also has it's own packaging mechanism.

Liberay = ESA

Karaf = Feature

Liferay = LPKG

Pure OSGi = Subsystem

However, the distro approach makes any of those approaches doable by first:

- installing everything you need in the container using container tools

- then make a distro

- then use that to resolve

- then from the resulting bundles build your package

It's still the process documented. Of course it didn't get into the specifics of each container but the result is still correct.

- Ray

[Raymond Auge]

On Wed, Dec 6, 2017 at 6:31 PM, Raymond Auge <raymon...@liferay.com> wrote:

On Wed, Dec 6, 2017 at 6:26 PM, Milen Dyankov <milend...@gmail.com> wrote:

 

The from the resulting bundles... make a feature!

 ... and than add that feature to the build + configuration to generate custom Karaf distribution. 

That is exactly why I said "will get you half way there" :) 

My point is that Karaf is not special here... Any container likely also has it's own packaging mechanism.

Liberay = ESA

Liberty

[Milen Dyankov]

My point is

 - Karaf features are actually "resolvable" and much like bundles can depend on each other. That is not how LPKG works AFAIK. I don't know about ESA and Subsystems

 - Karaf allows you to build custom distributions - that is, get only some parts of it and assemble a runtime. That is different from deploying to a pre-installed environment like Liferay or Liberty. 

And again, I'm not saying the bnd resolver is not helpful. Quite the opposite in fact. Just that one need to keep in mind 

 - the list of available bundles to use for resolving may only be available as interdependent features which bnd does not understand at the moment.

 - the assumption you have a runtime into which you can deploy an agent may not hold true for Karaf

Best,

Milen

[Raymond Auge]

I think you have missed the point again.

I'll try to clarify.

On Fri, Dec 8, 2017 at 2:03 PM, Milen Dyankov <milend...@gmail.com> wrote:

My point is

 - Karaf features are actually "resolvable" and much like bundles can depend on each other. That is not how LPKG works AFAIK. I don't know about ESA and Subsystems

 - Karaf allows you to build custom distributions - that is, get only some parts of it and assemble a runtime.

Once you've done that... create the distro! This is no different that in Liberty installing a bunch of extensions from the container manager, or in Liferay installing a bunch of LPKGs from the marketplace.

Once you've done all that, run the distro command.

 

That is different from deploying to a pre-installed environment like Liferay or Liberty. 

It's really not! See my comment above. (open|Websphere)liberty and Liferay have the exact same behaviour as Karaf in this regard. Like the documentation says:

> What you need to bear in mind is that the distro file needs to be regenerated each time the target container changes in any significant way otherwise you won’t get the real state of the system needed to resolve against.

- Ray

[David Jencks]

I’d like to reinforce what Raymond says.

Sent from my iPhone

On Dec 8, 2017, at 12:10 PM, Raymond Auge <raymon...@liferay.com> wrote:

I think you have missed the point again.

I'll try to clarify.

On Fri, Dec 8, 2017 at 2:03 PM, Milen Dyankov <milend...@gmail.com> wrote:

My point is

 - Karaf features are actually "resolvable" and much like bundles can depend on each other. That is not how LPKG works AFAIK. I don't know about ESA and Subsystems

I don’t know about lpkg. Spec subsystems/esas and liberty specific subsystems (I don’t recall what they are called) import and export capabilities and requirements, so they very definitely require the use of the resolver to install. You can nest isolation regions which is a sort of dependency, but as I recall there’s no direct analogue of the mistake of require-bundle.

 - Karaf allows you to build custom distributions - that is, get only some parts of it and assemble a runtime.

Once you've done that... create the distro! This is no different that in Liberty installing a bunch of extensions from the container manager, or in Liferay installing a bunch of LPKGs from the marketplace.

Once you've done all that, run the distro command.

To be more explicit, liberty allows you to make custom distributions.

David Jencks

[Milen Dyankov]

This discussion becomes to look a bit like "Q: How do I do that in Maven? A: Use Gradle!" ;)

I'm not questioning the power of bndtools nor the fact it makes perfect sense to use it. It's just there are many different ways to build OSGI projects. I personally prefer to see the tools supporting different strategies rather than trying to convince everyone to change their approach to match what given tool is capable of doing. But that's just me apparently. 

Best,

Milen 

[Timothy Ward]

This discussion becomes to look a bit like "Q: How do I do that in Maven? A: Use Gradle!" ;)

I’m not sure that I follow - there are lots of Maven tools that users can use. The correct way to resolve against a custom runtime would still be to create a distribution definition - that’s the thing that you ned to resolve against.

Tim

[Milen Dyankov]

I'm sorry, Maven- Gradle thing was meant to be used only as a metaphor. I'll try to explain one more time even though I'm pretty sure my point of view will hardly make sense to the people in this list. 

Karaf developers are used to use features. So in terms if resolving, the question they'd have is "Given those features and my bundle, what other features I need to install?" That is a question bnd/bndtools can not answer. Of course if you are bnd/bndtools user and not Karaf user it's very easy to think "but it will tell you which bundles to install" which is the same and perhaps even better. That may or may not be the case (because features can have additional data like configuration for example). But that is not my point, my point is this is basically the same as saying "but you can use Gradle instead of Maven" or in more general form "don't use your tool, use ours". 

We can argue forever what makes more sense, what is best, etc. I really don't want into that discussion. I was just responding to Ray's "the problem is any different than with any pre-build container". I do believe it is a bit different due to the nature of the features, the fact there is feature resolver in Karaf and the widely used approach to build custom Karaf distributions. I do agree one can give up on using those and go for bnd/bndtools approach but that is not the same thing. It would be much better if bnd could be used to actually help with Karaf specific tasks, which IMHO is what Christian was aiming for in his original mail.

Best,

Milen

 

http://about.me/milen

[Tim Ward]

And my point was that if you want to make a Karaf feature then you can do 99% of that using using bnd’s resolving capabilities and a target distribution. 

The bit that you’re left to do is turning a list of runbundles into a feature description, at which point I believe there are Karaf tools to help you. 

Fundamentally a Karaf Feature is a proprietary descriptor, so I’m not sure how much more help Bndtools can be. It’s not like the situation is different for any other form of packaging. 

Tim

Sent from my iPhone

[Peter Kriens]

It is actually quite easy to do in bnd, the plugin architecture is in place. Just need some company to  fund it ..

Kind regards,

Peter Kriens

[Milen Dyankov]

"if you want to make a Karaf feature" - well this is the perception problem right here. But I don't want to simply make a Karaf feature. I want to know which other Karaf features that feature needs. Much like I want to know which other bundles my bundle needs as opposite to discovering what classes/packages it needs and then putting them all into my bundle. The misunderstanding seems to be that from bnd standpoint, feature is noting more than a way of packaging or as you put it a "proprietary descriptor". That is almost like saying that a bundle in nothing more than a container for classes/packages and pretend requirements and capabilities does not exists. 

[Timothy Ward]

"if you want to make a Karaf feature" - well this is the perception problem right here. But I don't want to simply make a Karaf feature. I want to know which other Karaf features that feature needs.

And this is the point where this stops being an OSGi question and starts being a set of Karaf questions. 

• How do I index a set of features so that I know what they provide and what they require? Tools exist to do this for bundles, but I don’t know about features. Without this tool it is not possible to make progress as the feature is just some opaque blob that we can’t understand.
• Does my Karaf feature really need other features? I know that a dependency on a feature can be expressed, but my guess is that a Karaf feature dependency isn’t really a feature dependency at all, but a shorthand for pulling in a bunch of bundles which provide the capabilities required by the bundles you want to deploy. This sort of require-by-name model is messy at best, and fails to take into account that other people may already be providing the capabilities that you need. It’s effectively a legacy way of managing bundles from before resolving was possible. Given all this, is expressing a feature dependency really the right way for me to write my feature, or should I be leaving dangling requirements and trying to resolve instead?
• If I want to calculate a Feature dependency graph how can I give that to the resolver? To answer this question then you need someone to do the work to convert between the non-standard Feature dependency format and a format that the resolver understands (namely OSGi resources). If and when a tool is created to represent a Karaf feature as a resource in a repository then Bndtools will be able to feed it to the resolver just like anything else.

Now the original question that started this thread involved someone who wanted to use bndtools for developing some bundles which they wanted to deploy into a custom Karaf runtime. This is where most of the advice started from. If you start from the perspective of wanting to use bndtools then the consensus result of asking these questions is that:

1. I should be setting up a Karaf runtime with the features that I want (step one of indexing a set of features)
2. I should create a standard index of the resulting runtime, known to many as a creating a distribution. (step two of indexing a set of features)
3. I should resolve my feature/application against this runtime. If I am unhappy with the size of my feature then I can adjust the set of installed features in my custom distribution and try again.
4. I can then choose to deploy my new feature either as a feature (which I have to create myself in some way) or as a set of bundles (which is closer to what bndtools is optimised for).

This is why Ray, David and I have all suggested this as a way of trying to develop with Karaf features. It provides a reasonable development experience, and works within the limits of bndtools' support for proprietary external environments like Karaf. We are not claiming that this is the only way to develop for Karaf, nor that this is necessarily the best possible way to develop for Karaf. What we are saying is simply that this is what we would do, using bndtools, in the original poster’s situation.

The misunderstanding seems to be that from bnd standpoint, feature is noting more than a way of packaging or as you put it a "proprietary descriptor". That is almost like saying that a bundle in nothing more than a container for classes/packages and pretend requirements and capabilities does not exists. 

No, a feature really is a proprietary descriptor. A feature is not an open standard (proprietary) and it is metadata describing something (a descriptor). A bundle manifest is also a descriptor (albeit a standard one) which describes a set of packages to be deployed. Furthermore I am not ignoring requirements and capabilities at all. As I say above, if you want map Karaf features into a requirements and capabilities model that bnd can understand then they can be resolved by bnd without issue, if you don’t want to do that then fine, but there’s not much that Bnd can do to help you.

Tim

[Milen Dyankov]

Tim,

conceptually speaking I mostly agree with you, Ray, David and everyone else with bnd-first approach. Also, I'm NOT suggesting nor expecting that bnd/bndtools should care about Karaf and give it any special treatment. 

There were 2 "requirements" in the original mail:
 - "... automate starting karaf when pressing debug ..."

 - make the export consider Karaf specific things 

It is clear to me that both of those need to be build on top of bnd/bndtools by people who care more about Karaf than bnd/bndtools maintainers.

The current discussion started a year after the original post, with the question "has there been any progress regarding this subject?" which was answered with "the problem is any different than with any pre-build container". I should have simply responded with  "No, there has been no progress in this subject! You still can not do those things with bnd/bndtools because AFAIK no one has build such extension/plugin yet." Instead I made the mistake of trying to explain why it is different. This naturally resulted in "don't expect any special treatment for Karaf" type of reaction. That standpoint is totally understandable and I have NO problem with that. All I wanted was to make it clear for the future readers of this thread that there is no Karaf specific solution in bnd/bndtools as of now.

Best,

Milen