How to use non-OSGi libraries with maven dependencies with bndtools? Beginner's questions...

[jlnho]

Hi all,

I'm pretty new to OSGi and bndtools and still feeling rather clueless, so I apologize if some of my questions don't really make sense.

I'm trying to build a small demo application to learn my ways around all the tools and concepts.

So far I've managed to install bndtools and create simple "Hello, World"-style applications. All good. But every time I want to start on something more serious I will find myself in a situation where I need some library that is not an OSGi bundle and that has a bunch of maven build-time dependencies. How can I use such libraries in my bndtools projects with minimum hassle? 

Here's a concrete example: I want to use influxdb-java which is not an OSGi-bundle itself has three other dependencies. I can download the plain jar and add it to the buildpath for those bundles that need access to it, but when I try to start such a bundle I will get errors because the aforementioned dependencies aren't met. Does this mean that every time I need some non-OSGi library I will sort of have to manually pull all the dependencies together and wrap everything in an OSGi bundle? Isn't there some easier way to do this?

Any help would be much appreciated. Thank you!

Julian 

[BJ Hargrave]

Have you looked at enRoute? http://enroute.osgi.org/ It includes several tutorials to get you started with OSGi concepts and bndtools.

--
You received this message because you are subscribed to the Google Groups "bndtools-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bndtools-user...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

--
BJ

[jlnho]

Yes I have. I also read some bndtools tutorials and FAQs like this one. But I couldn't find any answers to my question.

[jlnho]

A note on OSGi enRoute: This tutorial page does address my problem, but the solution contains exactly the things I'm trying to avoid: manual dependency hell!

On Wednesday, December 2, 2015 at 6:15:57 PM UTC+1, BJ Hargrave wrote:

[Peter Kriens]

Yeah, life sucks, as developers we are caught between a rock and a hard place. We can automatically drag in dependencies like maven does. A needs B,C. B needs D, E, C. D needs … you get the point. This model is popular because it is simple and shifts the burden to later. This is good if later is on the other side of the wall because then YOU win. However, when you look at the overall picture this model tends to create much larger systems. Complexity increases exponential with the number of parts so even dragging in one unneeded dependency can wreak significant havoc. That is, the other side of the wall is not so happy then when later hits the fan.

Worse, it is unavoidable in this model to get into the diamond version problem. If A needs B, C and B and C need D then inevitably you are, well, in trouble. Yes, afaik OSGi is the only runtime in the world that can handle this case by verifying uses constraints and making sure parties in different class spaces cannot communicate. Fantastic technology! However, it still sucks to be in that place because you can very easily end up with a split brain.

The best model that we found treats the building phase of the components very different from the construction of the final runtime. Good components have very few dependencies that are also very explicit. That is, these dependencies are a primary design concern of the component. 

The runtime is different. It tends to have a large number of dependencies because it aggregates the dependencies of all its constituent components. For this, we still need to have a curated repository to make sure we do not end up with a virus, but we could use help.

The only thing that we’ve found to work is to do the work upfront. When you build a skyscraper you spent an insane amount of time making sure your foundation is straight because any small deviation is going to cost you dearly at the 42nd floor. In almost all professions it is the preparation phase that defines later success. Software is no different.

In object oriented technology we ran into such a problem. In the 80’s and 90’s we freely used object references: our freedom should not be restricted! OTI (80’s & 90’s, Smalltalk, later was at the roots of Eclipse) was build around letting the computer figure out what parts were needed for an application by looking at the transitive dependencies. Easy! It was only when we learned that huge transitive dependency graphs made it virtually impossible to reuse smaller parts that someone came up with interfaces. Interfaces broke the transitive dependency graph and suddenly we could actually make components that could be reused in several different places because they were less coupled. 

It takes some time to see this for most people, but the transitive dependency model used in Maven is identical to the OO object reference model of the 80’s, the granularity is now an artifact instead object. It has the exactly the same problem.

To break this transitive model, OSGi advocates the service model, which acts as the Interface did for Java but now for components. In this model, you create cohesive components that are coupled through services only. A component can provide or consume services. In practice this means your component has very few dependencies wrt to the old world. Since services use strict API their dependencies are very limited. 

These limited number of dependencies must be manually designed because if you’re not careful the component will let itself go and quickly becomes a thick fat unusable sack of bits. 

In the application you combine all the needed components and construct a runtime. Though we could easily automate that via JPM there are a number of catches. First, you need an implement for the  Configuration Admin service, which one should we choose? Knopflerfish, Apache Felix, Equinox, ProSyst, etc? Second catch is optionality, lots of components have options.

Obviously these decision MUST be manual because the computer cannot know what kind of system you want to build. Reputation, curation, footprint, performance, the number of other services it drags in are all trade offs. The computer can help with by showing the right information but the decision is clearly in the realm of the architect. 

A few years ago Graham Charters from IBM came up with the Modularity Maturity Model [1]. An organization that understand that dependencies are a concrete design artifact and not a consequence of prior decisions is the difference between the levels 3 and 4.

I can understand the fear of handling your dependencies manually if you come from a Maven world. I hope I elucidated why systems based on transitive dependencies tend to become a nightmarish humongous brittle block of rigid bits. The sheer number of dependencies managed by many maven projects is daunting, the idea doing this by hand is rightfully scary. However, there are several things that make handling dependencies in an OSGi world significantly lighter. Since we depend on service API dependencies we have many fewer dependencies and we can use semantic versioning with extensive support from tools like Bndtools. In OSGi, dependencies have lubrication because an upstream component can change without affecting any of its downstream clients as long as the change in the API is compatible. In a rigid transitive dependency graph you must change the pom of EVERY downstream project to get access to it. Until you’ve truly experienced this in real projects it is hard to feel how much of a difference that makes.

What is extremely hard to get for a lot of people is that you should not try to manage that large set of dependencies, you just should not have that many dependencies! It is very hard to accept for many that in a lot of systems dependencies are dragged in that are actually never executed. And remember, complexity is exponentially proportional to the number of parts.

That said, you rightfully point out the problem that in the existing world there are a lot of projects that do not work that way. I recognize your pain that you see this cool project but then you try to use it and it drags in a world of dependencies. Though you can wrap those, I tend to stay away from those projects like the plague because they are not proper components and it is usually a lot of work to make them into one. If the project is really important to me then I tend to create a service that abstracts the way I want to work (and not more) with that project and I put the project in a bundle with all the transitive project dependencies providing just this service. Once you really understand OSGi, you will see that with Declarative Services and Configuration Admin make the whole thing surprisingly little effort, a true example of the power of modularity.

Long story. If you feel that the transitive dependency model works for you then OSGi might be a step too far; we’ve gone out of our way to kill it because we are convinced that it creates large and unwieldy systems. However, I’ve told this story untold times. Few people get it immediately, for most it takes time because it is one of those things you must ‘feel’ and not just rationally understand. (The easiest solution is to just assume I am an idiot. We then see you once you experience the brick wall: all development time is spent on maintenance.)

To answer your question. Avoid projects with large transitive dependencies and look for real OSGi components. The exact problem of having to manually drag in all these transitive dependencies for what is supposed to be a reusable component is a sign of bad quality. Be glad you see it early.

Hmm, I guess I just wrote another blog :-)

Hope this helps, kind regards,

Peter Kriens

[1]: http://www.slideshare.net/mfrancis/ibm-sponsorship-keynote-towards-a-modularity-maturity-model-graham-charters

[jlnho]

Thank you so much for your elaborate answer! I agree with pretty much everything you said. 

This part essentially describes what I'm trying to achieve right now: "If the project is really important to me then I tend to create a service that abstracts the way I want to work (and not more) with that project and I put the project in a bundle with all the transitive project dependencies providing just this service."

I'm trying to create a bundle that wraps all the dependencies needed for the non-OSGi component and exposes the functionality I need (and nothing more) via a simple service API. I guess I will just have to get used to the fact that this means there will be lots of manual fiddling involved every time I want to do this.

[Przemysław Wesołek]

Hi Julian,

I really feel your pain. I've been OSGi-ying my server application half a year ago and it was not all roses...

As an example, which might help you, I used RESTEasy, which is not OSGI-fied. I tried a few options, including embedding the library in one of my bundles or making it a proper bundle with Felix's popular maven-bundle-plugin or EBR (https://github.com/eclipse/ebr). I ended with writing my own based on https://github.com/metacosm/resteasy-osgi-bundle -- A snapshot is at https://gist.github.com/jest/1d58eca84faac08ad942 although comments might be a bit inaccurate).

Currently, when I'm missing some libs during development time (i.e. on the buildpath), I often go to JPM4J. It has a lot of bundles, also autobuilded from Maven Central. For example, here's influxdb-java: http://jpm4j.org/#!/p/org.influxdb/influxdb-java.

Unfortunatelly, JPM4J has to be searched manually, i.e. the bnd plugin doesn't implement searching for the bundles on-line: every bundle has to be added by hand, by drag-and-dropping the version icon to the repository in Bndtools. So it might be a bit of manual work, if the bundle you want uses external packages in their API (i.e. as a superclass, parameter type or return type): then you have to follow such dependency and "drag it" from JPM4J as well. Also, when such bundle has transitive dependencies you would need to resolve them before you run the code.

There's been a similar question to yours on this group a few days ago, from sarcobe. In his answer Tim Ward gave a pretty nice summary of the usage of new bnd-indexer-maven-plugin which IMO might be a hit when released.

There's also a Bndtool's bundle-hub (https://github.com/bndtools/bundle-hub) which is a nice repo to add, because it is indexed on your computer and you can use it for resolving runtime dependencies (e.g. in bndrun files, via "Resolve" button).

Surprisingly for me, in spite of the OSGi age, the environment around it is pretty vivid and always moving forward. This is of course a good thing, but the side effect is that a lot of information is either outdated or only in the heads of people. So searching and asking is a key.

If you want to read something more in depth, I always find moi.vonos.net author's articles very in-depth but very precise and clearly written. He also wrote some less advanced stuff about OSGi, e.g.

• http://moi.vonos.net/2013/03/osgi/ - if you want to know "how it works"
• http://moi.vonos.net/java/osgi-depres/ - this one is more advanced but closely related to your question about bundle finding and repositories

Sorry for going a bit off topic, but I found your question a good opportunity to write up my modest experience.

Przemek

[Kamil]

I know this topic is rather old, but it was the first one I found in Google.

For future readers - maybe this could help a little bit: https://groups.google.com/forum/#!topic/bndtools-users/W8XfUVCbR_E