Shock! OSGi framework loads jar without OSGi metadata!

[Chris Rankin]

Hi all,

There once was a jar that had obviously never been touched by bnd, but Gradle had added it to my ZIP of bundles anyway. I fed this ZIP into an OSGi framework, one jar at a time...

I was expecting the OSGi framework to reject the offending jar's InputStream immediately as invalid, but it didn't. The resulting bundle had a version of 0.0.0, no symbolic name and must presumably be considered to be highly toxic - particularly to OSGi's resolving process. Fortunately, it only contains a few Google annotations...

Is OSGi supposed to tolerate such things as plain ol' jars? I thought the Bundle-SymbolicName header was mandatory! (OK, obviously mandatory for bnd. But apparently only "extremely desirable" for an OSGi framework? That seems weird!)

Can anyone offer any enlightenment please?

Slightly confused,

Thanks,

Chris

[BJ Hargrave]

It was treated as a Release 3 bundle. But it is basically dead since it exports no packages and has no Bundle-Activator. 

--
You received this message because you are subscribed to the Google Groups "bndtools-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bndtools-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bndtools-users/868d3d0a-3161-47b4-825d-81cba6968cd9n%40googlegroups.com.

--

BJ

[Chris Rankin]

On Thursday, 22 October 2020 at 01:35:38 UTC+1 BJ Hargrave wrote:

It was treated as a Release 3 bundle. But it is basically dead since it exports no packages and has no Bundle-Activator. 

Hi, thanks for replying.

Yes, the jar is loaded but sadly useless :(. Which means I will need to get creative and "OSGify" any such jars before packing them into my ZIP file. Obviously I will have no choice but to tell bnd to export every single package inside the jar in cases like these. I actually have some code to do that already which just scans for .class files inside the jar file and injects their package names as an "Export-Package" instruction. ("Export-Package: *" did NOT do what I had hoped here....!) However, that was just for one jar, whereas this time it would need to be for N jars, implying I would need to create N new Bundle tasks after Gradle has resolved the configurations but before finalizing its task graph. (Ewwww!!)

It occurs to me that it might be worth writing a brand new Gradle task to do this that invokes bndlib directly. Would you recommend this approach please? Is there a (stable) "public API" for such usage, or would I just be creating a maintenance nightmare for myself as Bnd evolves? (Obviously still hoping that N dynamically-created Bundle tasks will work here..)

Thanks,

Chris

[BJ Hargrave]

There is already a bnd wrap command. Also see https://bnd.bndtools.org/chapters/390-wrapping.html.

Normally wrapping is done once and placed in a repository. I don't think you want to wrap dependencies in every build. This is why we do not have maven and gradle tasks for wrapping.

--
You received this message because you are subscribed to the Google Groups "bndtools-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bndtools-user...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bndtools-users/deceaac4-698a-4318-b5a3-b1b6c7849a34n%40googlegroups.com.

--

BJ

[Chris Rankin]

On Friday, 23 October 2020 at 16:30:35 UTC+1 BJ Hargrave wrote:

Normally wrapping is done once and placed in a repository. I don't think you want to wrap dependencies in every build. This is why we do not have maven and gradle tasks for wrapping.

Hi, thanks for replying, and for pointing me to the wrap command. Unfortunately (for me) I suspect I really do need to be wrapping jars like this because I need to automate my build process so that other people can use it without their heads exploding :-(. However, the wrap command has reminded me that my jars will need optional imports as well as version=0 on all their exports, so that's useful. And THEN we will require some way of making **** SURE that this toxic little bundle can only be resolved against a well defined set of other bundles, to preserve the integrity of the framework.

Some kind of padded cell with a good firm lock will help.

And once I've been safely locked away, the other team will deploy their OSGi sandboxing solution!

Thanks,

Chris

[Chris Rankin]

On Friday, 23 October 2020 at 17:40:20 UTC+1 Chris Rankin wrote:

However, the wrap command has reminded me that my jars will need optional imports as well as version=0 on all their exports

Actually, maybe I don't need "version=0" after all (having now fixed an error in my Gradle build).But I'll keep the "optional" imports :-)

Cheers,

Chris

[Peter Kriens]

Wrapping is a mysterious art. There is no way you can reliably make this work without human introspection. If you can, it is useless since you incorporate all problems of the flat class path. I.e. you buy big money for OSGi but then don't use its primary features that give lots of benefits.

Kind regards,

Peter Kriens

--
You received this message because you are subscribed to the Google Groups "bndtools-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bndtools-user...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bndtools-users/92824cf2-0b44-4df1-a6a0-c58e4de314dfn%40googlegroups.com.

[Chris Rankin]

On Saturday, 24 October 2020 at 11:20:03 UTC+1 Peter Kriens wrote:

Wrapping is a mysterious art. There is no way you can reliably make this work without human introspection. If you can, it is useless since you incorporate all problems of the flat class path. I.e. you buy big money for OSGi but then don't use its primary features that give lots of benefits.

Yes, which is why auto-wrapping can only be a "last line of defence" for things like com.google.errorprone:error_prone_annotations which we cannot prevent from being dragged in as transitive dependencies of things like Guava. (And this is despite Guava itself being an OSGi bundle!) Obviously the thing that Guava is likely to be a transitive dependency of is built by a human developer and bundled properly with bnd.

I think the basic idea is that a developer will give us a parcel on N bundles, some of which may have been auto-wrapped, and which s/he already knows work together correctly. The framework will then force these N bundles to resolve only against each other and a carefully selected API bundle. I imagine that this could result in a lot of duplication too... but I think we're fine with that. And it's still early(-ish) days.

Cheers,

Chris

[Peter Kriens]

I understand that model, Netflix is doing something similar. 

The problem is that you need to properly _analyze_ that set to make work reliable. You can quickly get to 98% percent working ok but the remaining 1% is going to cost you sooooo much pain ... Mainly because you're users are going to have no experience with the problems.

The beauty of Java is that it has a very decent formal specification and that the byte code allows you to an (almost) full transitive analysis.

Kind regards,

Peter Kriens

--
You received this message because you are subscribed to the Google Groups "bndtools-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bndtools-user...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bndtools-users/38381063-02f1-4ff6-a4bf-b92b6e1d6128n%40googlegroups.com.

[Chris Rankin]

On Saturday, 24 October 2020 at 14:11:43 UTC+1 pkriens wrote:

The problem is that you need to properly _analyze_ that set to make work reliable. You can quickly get to 98% percent working ok but the remaining 1% is going to cost you sooooo much pain ... Mainly because you're users are going to have no experience with the problems.

The beauty of Java is that it has a very decent formal specification and that the byte code allows you to an (almost) full transitive analysis.

I take your point about the potential problems with the "user experience"... :-).

Bnd has a fine "Resolve" Gradle task which should be able to validate that the user's set of N bundles all wire together correctly. I think this should be able to provide the kind of "full transitive analysis" of which you speak.

In fact, considering that Guava clearly resolves already without com.google.error-prone etc being bundles in the first place, it's possible we don't need to wrap all non-bundles automatically either. We could probably restrict that to just those dependencies which are explicitly added to some kind of "bundleImplementation" Gradle configuration. And then we'd tell users that they would only need to use the "bundleImplementation" when their "resolve check" has failed because something isn't a bundle that actually needs to be.

All of which would be more for Mr Gradle Bunny to implement,  of course. Oh well...

Thanks,

Chris

 

[Peter Kriens]

On 24 Oct 2020, at 17:14, Chris Rankin <rank...@gmail.com> wrote:

On Saturday, 24 October 2020 at 14:11:43 UTC+1 pkriens wrote:

The problem is that you need to properly _analyze_ that set to make work reliable. You can quickly get to 98% percent working ok but the remaining 1% is going to cost you sooooo much pain ... Mainly because you're users are going to have no experience with the problems.

The beauty of Java is that it has a very decent formal specification and that the byte code allows you to an (almost) full transitive analysis.

I take your point about the potential problems with the "user experience"... :-).

:-)

Bnd has a fine "Resolve" Gradle task which should be able to validate that the user's set of N bundles all wire together correctly. I think this should be able to provide the kind of "full transitive analysis" of which you speak.

Yes. But only when the metadata is correct. With the customers I work with we ensure that we do not use the general shitty jars from the enterprise space that do not seem to care what they depend on. Which is generally the kitchen sink. 

OSGi is a wonderful technology but it is not for free. The biggest problem has always been that OSGi tells people their baby is ugly and they do not take well to  this. However, if you play by the rules it has HUGE benefits.

In fact, considering that Guava clearly resolves already without com.google.error-prone etc being bundles in the first place, it's possible we don't need to wrap all non-bundles automatically either. We could probably restrict that to just those dependencies which are explicitly added to some kind of "bundleImplementation" Gradle configuration. And then we'd tell users that they would only need to use the "bundleImplementation" when their "resolve check" has failed because something isn't a bundle that actually needs to be.

I think you should forget OSGi and just use the underlying technology to analyze an app class path the user uses. This analysis can then tel the user that  he's  got X dependencies that are not  found on the class path and  if it is ok that we run that time bomb. That way it is her fault if things blow up.

Kind regards,

Peter  Kriens

All of which would be more for Mr Gradle Bunny to implement,  of course. Oh well...

Thanks,

Chris

 

--
You received this message because you are subscribed to the Google Groups "bndtools-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bndtools-user...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bndtools-users/1ceed6f1-1f0c-439d-8ae0-cb50d4d9fd60n%40googlegroups.com.

[Chris Rankin]

On Saturday, 24 October 2020 at 18:33:46 UTC+1 Peter Kriens wrote:

Yes. But only when the metadata is correct.

This isn't the first suggestion I've seen that third-party OSGi meta-data may actually be broken, as well as missing entirely. We're going to need to be able to handle both cases. But that implies repackaging our N jars into one fat jar, and then turning that into a bundle ourselves. (Aagh!) Although it's not too scary from our framework's point of view because we want the N jars to resolve against each other anyway.

I think you should forget OSGi and just use the underlying technology to analyze an app class path the user uses. This analysis can then tel the user that  he's  got X dependencies that are not  found on the class path and  if it is ok that we run that time bomb. That way it is her fault if things blow up.

Isn't this step called "testing that the code actually runs before using bnd on it"? I think that was always part of the plan somewhere :-D.

Thanks, you've given me a lot to think about. I'm sure the rest of the team will appreciate your insights.

Cheers,

Chris

[Fr Jeremy Krieg (Home)]

Hi Chris,

Unfortunately, broken OSGi metadata seems to be extremely common (perhaps even the rule rather than the exception!). Things like packages that should be optional marked as mandatory, missing requirement declarations for ServiceLoader, split packages that make bundles impossible to resolve, etc. We have a couple of ideas for new features in bnd that can fix metadata in 3rd party dependencies rather than forcing you to generate it from scratch. A matter of time and resources. 

Blessings, 

Fr Jeremy 

--
You received this message because you are subscribed to the Google Groups "bndtools-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bndtools-user...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bndtools-users/035222fd-9962-4f96-83f8-42fa338ae759n%40googlegroups.com.

[Chris Rankin]

On Sunday, 25 October 2020 at 02:10:38 UTC Fr Jeremy Krieg wrote:

Unfortunately, broken OSGi metadata seems to be extremely common (perhaps even the rule rather than the exception!). Things like packages that should be optional marked as mandatory, missing requirement declarations for ServiceLoader, split packages that make bundles impossible to resolve, etc. We have a couple of ideas for new features in bnd that can fix metadata in 3rd party dependencies rather than forcing you to generate it from scratch. A matter of time and resources. 

Oh dear, I think I can smell my goose cooking... :-(. Thanks for the "heads up" about ServiceLoader though. I am assuming you are referring to this:

https://bnd.bndtools.org/chapters/240-spi-annotations.html

which means that I'll need to make the @ServiceConsumer and @ServiceProvider annotations automatically available to users. But that's still only half the battle because we'll need to guide people in their proper usage too. Or possible discourage their use in the first place. (We are fortunately in a position to be able to define our own "application space" to a certain extent.)

We can probably warn users if their bundle contains a META-INF/services/ folder but no "osgi.serviceloader" capability though. Or does bnd do that already?

Cheers,

Chris

[Jürgen Albert]

We should make this the new slogan for OSGi! :-)

@Peter: would you mind if we quote you on this? Maybe by putting it on
shirts and coffee mugs. Before anyone asks: Yes on Socks too! :-)

Am 24/10/2020 um 19:33 schrieb Peter Kriens:
> OSGi is a wonderful technology but it is not for free. The biggest
> problem has always been that OSGi tells people their baby is ugly and
> they do not take well to  this.

--
Jürgen Albert
Geschäftsführer

Data In Motion Consulting GmbH

Kahlaische Str. 4
07745 Jena

Mobil: 0157-72521634
E-Mail: j.al...@datainmotion.de
Web: www.datainmotion.de

XING: https://www.xing.com/profile/Juergen_Albert5

Rechtliches

Jena HBR 513025

[Peter Kriens]

> @Peter: would you mind if we quote you on this? Maybe by putting it on shirts and coffee mugs. Before anyone asks: Yes on Socks too! :-)

:-) Go ahead ...

It always flummoxes me if I see how messy most builds are on virtually all levels. It is a testament to Java that it so often works. But always cheers me up (and my customers) to see how many problems disappear when you clean things up by going to OSGi and not introduce a new mess by using bad or half baked bundles. There is a huge advantage in purity.

Kind regards,

Peter Kriens

> --
> You received this message because you are subscribed to the Google Groups "bndtools-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to bndtools-user...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/bndtools-users/83babb1c-aaf6-ca5e-af6b-8bbe760d50e4%40data-in-motion.biz.

[Hallvard Trætteberg]

Peter,

I teach OSGi and use bnd(tools) in a master class on "advanced" software design, and try to explain why a structured and "pure" approach is good. I use Eclipse as an example of a system with a scale and ecosystem where it makes a difference to be structured and think about modularity and extensibility in advance. But students have never taken part in developing systems where this is necessary, so they don't immediately appreciate it. Do you have any "stories" written up providing insights into cases where it mattered, that I can share with my students?

Regards,

Hallvard

To view this discussion on the web visit https://groups.google.com/d/msgid/bndtools-users/11CAB976-C903-4EF7-ABAD-BA425A0BDEE5%40gmail.com.

[Jürgen Albert]

Well, we have a few theories about that, because we see the same theme
with a lot of our customers as well. Mostly people accept what they have
been given and instead of fixing it, they prefer to bitch about it.
Usually the root cause of their problems, is declared the problem of
somebody else. Here you can take your pick, if it is the framework, the
tooling, the other team or the external consultant that left the project
ages ago. Take this with the internal or external pressure to get shit
done and a lack of knowledgable forces in an organisation to drive
change and you have your recipe for the shit we see often.

[Raymond Auge]

My company which I had been at for almost a decade by the time we discovered OSGi was mired in technical debt, thousands of features, a growing engineering organization breaking into teams and a very fragile plugin model.

Imagine 100 developers contributing to the same codebase without a modularity paradigm beyond JavaEE webapps model. We had devised over time a slew of _tricks_ (a.k.a. hacks) to make plugability work but it was starting to cause us problems and we were having huge issues just releasing our software trending to a greater than 2 year turnaround time for releases.

We needed a robust standard for plugins.

We needed robust isolation.

We needed reliable modularity so we could scale teams without blocking everyone.

Since then we have re-engineered our product to be about 80% OSGi (still runs inside a servlet container but that's not OSGi's fault).

We have improved our release cycle; though that benefit took a while to reach but we finally did recently.

Our engineering team grew to about ~300 developers.

We grew to around 15 separate, autonomous feature teams.

Some will argue that we could have done this some other way, but I fervently argue that without OSGi it would not have been impossible. OSGi enabled much if not all of this.

- Ray

To view this discussion on the web visit https://groups.google.com/d/msgid/bndtools-users/1fbfa468-af17-4ebb-fc88-e5cb15231460%40data-in-motion.biz.

--

Raymond Augé (@rotty3000)

Senior Software Architect Liferay, Inc. (@Liferay)

OSGi Fellow

[Mark Hoffmann]

I currently work for a customer, who already uses OSGi, as RCP and in a servlet container at backend side. They have really a lot of bundles and most of them are not really good in shape. Beside that no one really cares about their dependencies. This project is a good bad example for a modular monolith. Many developers in the teams dont care about OSGi at all. Its just there, they hate it, they don't really take profit of it. The whole team develops like in ordinary Java. And providing imports and exports is just a work, they don't really like.

I dont think, that this is only a result of just bad habbits. It really goes back to the company structure and culture. Non technical departments tend to spend money for their features not in technical improvements that are not obvious for customers. They often don't care about efficient development processes, well structured software architecture. This is something they expect from the development department, something that should already be there. But if it isn't, the technical dept, stay dept until the end of days until someone realizes that developers and business together create the product.

But this is not dedicated to OSGi alone. You can substitute these situation with any technology and end up with the same root cause ;-)

So, OSGi is overhead for many developers at this customer. And with Peters words: They have an ugly baby. But because it is already OSGi, its much easier to beautify it. But how to convince the developers, that they are on the right track. All they have to do is learn a little bit and try to understand whats happening inside OSGi. These discussions are in the end the hardest part.

Mark

Am 28.10.20 um 14:26 schrieb 'Raymond Auge' via bndtools-users:

To view this discussion on the web visit https://groups.google.com/d/msgid/bndtools-users/CAMm6HcASzxCuA3CefNmnOmJZLVmAH%2BpnygpEn0hjMoCN%2B7%3DSDQ%40mail.gmail.com.

-- 
Mark Hoffmann
M.A. Dipl.-Betriebswirt (FH)
Geschäftsführer

Tel:    +49 3641 384 910 0
Mobil:  +49 175 701 2201  
E-Mail: m.hof...@data-in-motion.biz
Web: www.datainmotion.de 

Data In Motion Consulting GmbH
Kahlaische Straße 4
07745 Jena

Geschäftsführer
Mark Hoffmann
Jürgen Albert

Jena HRB 513025
Steuernummer 162/107/05779
USt-Id DE310002614

[Peter Kriens]

Just imagine you're building a sky scraper of 150 floors. You think you can hack the foundation? Complexity is exponential. Small hacks and impure solution tends to strike back in quite a number of places and times. Usually the most inconvenient.

Unfortunately I do not have a story to share since most of this is under NDA and the culprits tend to listen here :-)

Why not let your students fix real bugs in open source projects in the class? That will quickly teach them how messy working software is. I made lots of mistakes over the last 20 years in bndlib because I thought it would remain a small program and did not want to use OSGi at the time. And you get free reviews from BJ! :-) Although bndlib is relatively straightforward since it has 0 (zero!) dependencies. A project like OpenHAB might be interesting.

I'd be more than willing to help to make teacher material with Bndtools. If there are other teachers out here that also us Bndtools for teaching CS students than they might also be interested?

Kind regards,

Peter Kriens

[1]: https://www.openhab.org/

> To view this discussion on the web visit https://groups.google.com/d/msgid/bndtools-users/D5D54B21-A39A-40ED-AF73-EA305166C246%40gmail.com.

[Peter Kriens]

I remember a very good story in this area is from Bezos. I think around 2005 Bezos personally send out a missive that no team was allowed to cross team boundaries _except_ with a formally defined REST API. This was the  result of years of growing pains where one team used a convenient customer table from another project. They created so many unmanaged dependencies that Bezos saw it would not scale to what he had in his mind. These unmanaged dependencies tend to either result in a freeze of a certain implementation despite having it designed as an implementation detail, or a compatibility break. Both bad. 

This probably was a 100+ million decision to implement it. A few years later Google panicked because they had not done this and ran into the expected problems. My estimate is that a significant number of those billions in Bezos' coffer from from this single decision.

Interestingly, most designers hated it I recall. 

Kind regards,

Peter Kriens

To view this discussion on the web visit https://groups.google.com/d/msgid/bndtools-users/69909b10-5d7c-4a0f-7428-60118fd42985%40data-in-motion.biz.