Does Bnd support OSGi SubSystems?

[Chris Rankin]

Hi,

Does the Gradle builder 5.1.2 plugin support exporting to an OSGi subsystem please? The documentation suggests that I just need use an Export task with this exporter type:

    osgi.subsystem.application

but Bnd is definitely not happy:

    error  : No exporter for osgi.subsystem.application

According to the OSGi spec, an ESA just seems to be a ZIP of bundles with an optional manifest, and so I didn't think I needed to use Bnd to create one. However, the SCR seems to be completely failing to launch/notice any components inside my subsystem's bundles and so I think my naive ZIP archive must be missing something. (Yes, the components work fine when installed not as part of a subsystem, so they are themselves correct.)

Any help here would be much appreciated.

Thanks,

Chris

[BJ Hargrave]

The subsystem exporters are in a different jar than bndlib. See the exporters project (https://github.com/bndtools/bnd/tree/master/biz.aQute.bnd.exporters) which includes the subsystem exporters.

So the documentation is not quite accurate. The bnd command includes the exporters jar and so it can do subsystems exports. But bndlib does not and bndlib is used by gradle and maven plugins. So to use the subsystem exporters there, you will need to configure the plugin, aQute.bnd.exporter.subsystem.SubsystemExporter, as a bnd plugin with a `-plugin` instruction and make sure the exporters jar is accessible via the class loader.

--
You received this message because you are subscribed to the Google Groups "bndtools-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bndtools-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bndtools-users/d83ad643-0bde-4029-9934-81dc965a294en%40googlegroups.com.

--

BJ

[Chris Rankin]

On Wednesday, 14 October 2020 at 14:03:24 UTC+1 BJ Hargrave wrote:

The subsystem exporters are in a different jar than bndlib. See the exporters project (https://github.com/bndtools/bnd/tree/master/biz.aQute.bnd.exporters) which includes the subsystem exporters.

So the documentation is not quite accurate. The bnd command includes the exporters jar and so it can do subsystems exports. But bndlib does not and bndlib is used by gradle and maven plugins. So to use the subsystem exporters there, you will need to configure the plugin, aQute.bnd.exporter.subsystem.SubsystemExporter, as a bnd plugin with a `-plugin` instruction and make sure the exporters jar is accessible via the class loader.

Hi, thanks for replying.

I have added the exporters jar to the classpath via the buildscript:

    buildscript {
         dependencies {
             classpath "biz.aQute.bnd:biz.aQute.bnd.exporters:$bnd_version"
         }
    }

And then added tha plugin line to my bndrun file:

    -plugin: aQute.bnd.exporter.subsystem.SubsystemExporter

Gradle's Export task now accepts the

    exporter = 'osgi.subsystem.application'

option. However, this exporter seems to have different input requirements from the other exporters because it now fails with:

error  : Cannot find /error/com.example.osgi.myapporg.osgi.service.cmorg.apache.felix.configadminorg.apache.felix.logbackorg.apache.felix.log;version=0 Not found in [export ...

which appears to be the concatenated contents of my bndrun file's  "-runrequires"  section:

-runrequires: \
     bnd.identity;id='com.example.osgi.myapp',\
     bnd.identity;id='org.osgi.service.cm',\
     bnd.identity;id='org.apache.felix.configadmin',\
     bnd.identity;id='org.apache.felix.logback',\
     bnd.identity;id='org.apache.felix.log'

It looks like this exporter doesn't expect to invoked via Gradle - for some reason.

Would I be correct in believing that OSGi subsystems are not widely used please?

Cheers,

Chris

[Raymond Auge]

Subsystems are not widely used.

- Ray

--
You received this message because you are subscribed to the Google Groups "bndtools-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bndtools-user...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bndtools-users/ced9b2e8-10a0-4b68-90dc-f1f9a92f904an%40googlegroups.com.

--

Raymond Augé (@rotty3000)

Senior Software Architect Liferay, Inc. (@Liferay)

[BJ Hargrave]

I am sure the exporter expects any referenced bundles to come from the configured repositories in the workspace (and bndrun file). And this would not include the FileSetRepository which is created for a resolve or execution operation.

And as Ray says, subsystems are not widely used. So there is not much experience in using the exporters in maven/gradle scenarios. Most usage is probably in the Bnd Workspace model.

On Wed, Oct 14, 2020 at 10:31 AM Chris Rankin <rank...@gmail.com> wrote:

--
You received this message because you are subscribed to the Google Groups "bndtools-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bndtools-user...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bndtools-users/ced9b2e8-10a0-4b68-90dc-f1f9a92f904an%40googlegroups.com.

--

BJ

[Chris Rankin]

On Wed, 14 Oct 2020 at 16:17, 'Raymond Auge' via bndtools-users <bndtool...@googlegroups.com> wrote:

Subsystems are not widely used.

- Ray

 

Ah, thanks. Suddenly the large bruise on my forehead makes sense. As well as the tumbleweed in the Aries user mailing list... My problem is that if I cannot get a viable subsystem running then I'm concerned that we'll try to reinvent them - and probably less well :-(. Can you recommend any resources beyond the OSGi spec please? I'm almost desperate enough at this point to check out the Aries source code and start println debugging!

Cheers,

Chris

[Raymond Auge]

We should probably start with what goal(s) you're trying to achieve. There are several w.r.t. subsystem type constructs so we should probably get on the same page :)

- Ray

--

You received this message because you are subscribed to the Google Groups "bndtools-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bndtools-user...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bndtools-users/CAK2bqV%2Btc%2B2MnwP0TnKRRxAQaU%3DEHnaT6YQACu%2BGOE%3DVSANQyw%40mail.gmail.com.

[Chris Rankin]

Hi,

I described my underlying problem on StackOverflow, which is probably a better forum than here on bndtools-users:

https://stackoverflow.com/questions/64339151/osgi-application-subsystem-is-active-but-its-components-are-not

Thanks,

Chris

You received this message because you are subscribed to a topic in the Google Groups "bndtools-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/bndtools-users/Llc9imaJcnI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to bndtools-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bndtools-users/CAMm6HcDA_QgP9OZiC_N0gV-L19b7BMN898bp68C8yGL0uAsvfA%40mail.gmail.com.

[Raymond Auge]

Is SCR impl inside the subsystem or outside?

- Ray

To view this discussion on the web visit https://groups.google.com/d/msgid/bndtools-users/CAK2bqVLGuH6R6%2BFDQ4-g8Z0AaWw%2B5tt0zZTFyy6v1RmtCvEt1A%40mail.gmail.com.

[Chris Rankin]

Outside. The subsystems are expected to be provided by external developers and only contain their own code (plus its dependencies). My understanding was that any requirement would be imported from the parent subsystem if it couldn't be satisfied internally. That is definitely the goal here anyway.

Cheers,

Chris

To view this discussion on the web visit https://groups.google.com/d/msgid/bndtools-users/CAMm6HcBmwVmcLGaUdH8EKk0YT9QwKk-ZHwdnJFHwnfM5SjT%2BoA%40mail.gmail.com.

[Raymond Auge]

If only Tom Watson was listening in... he'd know the finer details.

- Ray

To view this discussion on the web visit https://groups.google.com/d/msgid/bndtools-users/CAK2bqVLHeVY%2BE4wcEKCSi_LM4BPXB976X1AKsqaJoRj5wqowQw%40mail.gmail.com.

[Neil Bartlett]

On Wed, 14 Oct 2020 at 17:16, Chris Rankin <rank...@gmail.com> wrote:

Outside. The subsystems are expected to be provided by external developers and only contain their own code (plus its dependencies). My understanding was that any requirement would be imported from the parent subsystem if it couldn't be satisfied internally. That is definitely the goal here anyway.

That's almost certainly the problem. SCR is not just a dependency, it scans bundles for the `Service-Component` header. The Declarative Services specification does not describe any way for SCR to discover bundles inside a subsystem of the running framework, therefore your bundles will be invisible to it.

 

To view this discussion on the web visit https://groups.google.com/d/msgid/bndtools-users/CAK2bqVLHeVY%2BE4wcEKCSi_LM4BPXB976X1AKsqaJoRj5wqowQw%40mail.gmail.com.

[David Jencks]

I certainly thought I documented this when I implemented it, but it doesn’t appear to be currently documented.  IIRC you need to configure SCR with the ds.global.extender

flag set to true, then the single SCR will find components everywhere.  I believe I was talked out of making true the default value, but I don’t recall why; it’s what I’d expect.

David Jencks

To view this discussion on the web visit https://groups.google.com/d/msgid/bndtools-users/CAPr%3D90O3HK1UpWmjfgreHfnGGkZa9xzTJ-gv%3Ds7BjGYVdYzq2Q%40mail.gmail.com.

[Chris Rankin]

On Wed, 14 Oct 2020 at 17:40, David Jencks <david.a...@gmail.com> wrote:

Icertainly thought I documented this when I implemented it, but it doesn’t appear to be currently documented.  IIRC you need to configure SCR with the ds.global.extender

flag set to true, then the single SCR will find components everywhere.  I believe I was talked out of making true the default value, but I don’t recall why; it’s what I’d expect.

Oh my, that sounds ideal! Thanks, I will try that immediately. Actually, can you also say that on SO please? This will allow me to mark the question as answered.

Thanks again,

Chris

[Chris Rankin]

On Wednesday, 14 October 2020 at 17:40:37 UTC+1 djencks wrote:

I certainly thought I documented this when I implemented it, but it doesn’t appear to be currently documented.  IIRC you need to configure SCR with the ds.global.extender

flag set to true, then the single SCR will find components everywhere.  I believe I was talked out of making true the default value, but I don’t recall why; it’s what I’d expect.

Specifically, I have added this to the `-runproperties` inside my bndrun file:

    ds.global.extender=true

and this does the trick! Thanks, everyone.

Cheers,

Chris

 

P.S. For some reason, I was expecting to need to prefix this property name with "org.osgi.service." or "org.osgi.service.scr." in order for OSGI to recognise it as a configuration property for its SCR service.

[BJ Hargrave]

The property is a Felix SCR implementation feature. It is not part of the OSGi Declarative Services specification.

--
You received this message because you are subscribed to the Google Groups "bndtools-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bndtools-user...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bndtools-users/2a3ed3aa-4b17-4715-93e4-70f3d6193006n%40googlegroups.com.

--

BJ

[Chris Rankin]

On Wed, 14 Oct 2020 at 18:49, BJ Hargrave <b...@bjhargrave.com> wrote:

The property is a Felix SCR implementation feature. It is not part of the OSGi Declarative Services specification.

Ah, and the reason why this property is false by default is revealed - because it enables non-standard behaviour!

Thanks, that is good to know.

As an aside, I did experiment with obtaining a @Reference to the root's ServiceComponentRuntime service, adding it

programmatically to the newly installed subsystem and then starting the subsystem up. However, this wasn't enough to

activate component discovery. But now I'm wondering if this approach could have been made to work after all...

Cheers,

Chris

[David Jencks]

On Oct 14, 2020, at 11:41 AM, Chris Rankin <rank...@gmail.com> wrote:

On Wed, 14 Oct 2020 at 18:49, BJ Hargrave <b...@bjhargrave.com> wrote:

The property is a Felix SCR implementation feature. It is not part of the OSGi Declarative Services specification.

Ah, and the reason why this property is false by default is revealed - because it enables non-standard behaviour!

My recollection from a long time ago (the spec might have clarified this by now) is that the spec doesn’t say anything one way or the other about this, both settings were spec compliant.

Thanks, that is good to know.

As an aside, I did experiment with obtaining a @Reference to the root's ServiceComponentRuntime service, adding it

programmatically to the newly installed subsystem and then starting the subsystem up. However, this wasn't enough to

activate component discovery. But now I'm wondering if this approach could have been made to work after all…

How did you add the reference? I can’t imagine a way this could possibly work.

Anyway this is off-topic for bndtools…

David Jencks

Cheers,

Chris

--
You received this message because you are subscribed to the Google Groups "bndtools-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bndtools-user...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bndtools-users/CAK2bqVKMLUXW_Ogc5SFm5DkJBQj1gtTXWQVv1nTsyXOR%2Bn2-NQ%40mail.gmail.com.

[Chris Rankin]

On Wednesday, 14 October 2020 at 21:12:51 UTC+1 djencks wrote:

How did you add the reference? I can’t imagine a way this could possibly work.

    subsystem.getBundleContext().registerService(...)

And no, it didn't work :-). Thanks for the heads-up, I won't waste any more time investigating this.

 

Anyway this is off-topic for bndtools…

Indeed, but I did try and redirect it to StackOverflow ;-).

Cheers,

Chris

[Peter Kriens]

If you do not have a very strong mandatory binding required obligation to use subsystems I would not touch them with a ten foot pole. 

My preferred way of delivering code is via the fat jar approach. The set of bundles is then assembled using the resolver that can have a global view of the dependencies. Features, subsystems, etc., are all attempts at a partial delivery model where you share dependencies. This requires assumption about the runtime that are too easily violated or create contradictory constraints.

Subsystems is probably then the worst of all worlds. Although Java made a decent attempt, it is clearly not as isolating between bundles as we originally thought. The only proper isolation is the process. Subsystems dig an even deeper hole since the  isolation between subsystems is even less than between bundles and it requires a very complex implementation.

Although it was the original goal of OSGi to share in runtime, the software industry has been abandoning this model everywhere. If you download a mobile app today, it is often 60Mb because each app carries _all_ its dependencies. There is no Java delivered by Apple because each app must bring its own VM. The only thing you want to share are API to the environment. 

Although fat jars are not _efficient_, they are remarkably _robust_. Sharing is really hard and flash memory is really cheap. 

Kind regards,

Peter Kriens

--
You received this message because you are subscribed to the Google Groups "bndtools-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bndtools-user...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bndtools-users/ced9b2e8-10a0-4b68-90dc-f1f9a92f904an%40googlegroups.com.

[Chris Rankin]

On Thursday, 15 October 2020 at 08:15:15 UTC+1 Peter Kriens wrote:

If you do not have a very strong mandatory binding required obligation to use subsystems I would not touch them with a ten foot pole. 

My preferred way of delivering code is via the fat jar approach. The set of bundles is then assembled using the resolver that can have a global view of the dependencies. Features, subsystems, etc., are all attempts at a partial delivery model where you share dependencies. This requires assumption about the runtime that are too easily violated or create contradictory constraints.

Subsystems is probably then the worst of all worlds. Although Java made a decent attempt, it is clearly not as isolating between bundles as we originally thought. The only proper isolation is the process. Subsystems dig an even deeper hole since the  isolation between subsystems is even less than between bundles and it requires a very complex implementation.

Although it was the original goal of OSGi to share in runtime, the software industry has been abandoning this model everywhere. If you download a mobile app today, it is often 60Mb because each app carries _all_ its dependencies. There is no Java delivered by Apple because each app must bring its own VM. The only thing you want to share are API to the environment. 

Although fat jars are not _efficient_, they are remarkably _robust_. Sharing is really hard and flash memory is really cheap. 

Hi, thanks for replying.

That certainly puts the cat amongst the pigeons!

While "isolation" is one of our overall requirements, our intention behind using subsystems would be "composition" - i.e. needing multiple jars-of-jars to work with both a common core and potentially each other, and to continue working together over time, despite some of them containing different versions of a common dependency (e.g. Guava). And if one subsystem uses "foo-1.2.3.jar" internally then that would only matter when the subsystem itself is resolved, not when the subsystem is resolved against the rest of the system.

Your critique of isolation in Java could be read as a vote against OSGi in general, not just of its subsystems. And when you say (I'm not disagreeing, mind) "the only proper isolation is the process", that criticism would appear to extend to using JPMS as well. This is certainly something that we must consider.

FWIW I am a total OSGi n00b and know even less about JPMS... :-). This has all been an arduous learning experience for me.

Thanks again,

Chris

[Peter Kriens]

You are right that some of these issues go against OSGi. This is partly a process of learning over 20 years, new technologies that created new possibilities, and foremost the drop in computing & storage cost. 

My personal reason to still love OSGi, apart from being around forever, is mostly because I _love_ the development process. Just today, it is just amazing how easy it is to quickly create a new small runtime out of nothing with the resolver, and completely interactively develop an application that pushes the logs to Elastic Search. And then the best part is that I am sure it works in almost all cases since the tooling and OSGi metadata verifies things that no other environment checks, and therefore often gets wrong.

I am currently mostly working in embedded/gateways and that really renewed my love for OSGi. It is a relief to leave the messy world of enterprise code where the metadata is often wrong, modularity is a foreign word, dependencies are good so more is better, developers love to hack their way around in the class path, and they want to just put everything on the class path they feel like, and then it should all work. OSGi works brilliantly when you do things according to the book but it tells you very loud clear that your baby is ugly when you don't.

Anyway. I am not sure what you're use case is but hoping that you can throw a messy class path into a container and make it work is probably going to be problematic. If the apps really cannot each have their own framework, then one trick I sometimes use is to deliver the application/subsystem as a single bundle that contains all its dependencies. The bnd -conditionalpackage is very nice for this. I know Netflix uses this trick. As long as you keep the dependencies in private there is no chance of class conflicts.

Anyway. There are lots of solutions out there but subsystems are probably not one of them :-) If this is a commercial application and you need some advice then don't hesitate to contact me.

Kind regards,

Peter Kriens

-- 
You received this message because you are subscribed to the Google Groups "bndtools-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bndtools-user...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bndtools-users/bdd99d4f-07b9-4696-8544-3ec7e12edfbfn%40googlegroups.com.

[Chris Rankin]

On Thursday, 15 October 2020 at 13:14:25 UTC+1 pkriens wrote:

Anyway. I am not sure what you're use case is but hoping that you can throw a messy class path into a container and make it work is probably going to be problematic. If the apps really cannot each have their own framework, then one trick I sometimes use is to deliver the application/subsystem as a single bundle that contains all its dependencies. The bnd -conditionalpackage is very nice for this. I know Netflix uses this trick. As long as you keep the dependencies in private there is no chance of class conflicts.

Ironically, the only reason I started looking at OSGi subsystems in the first place was that it seemed to be what had evolved out of this 2010 blog post;

https://blog.osgi.org/2010/01/nested-frameworks.html

Initially, I was trying and failing to start a brand new OSGi framework inside an existing OSGi framework. However, the framework was unwilling to surrender its FrameworkFactory without a fight.

Cheers,

Chris

[Neil Bartlett]

If I may add my two cents... I tend to agree with Peter on a lot things, and deployment of applications or systems – especially those built by external developers, as in your case – is best done as an isolated unit with all dependencies provided. You should not have granular dependencies on artifacts outside of the deployment unit, and certainly do NOT try to share libraries between independent deployments... the miniscule gains in terms of install footprint are far outweighed by the complexities of versioning, managing when these shared libraries need to be upgraded or uninstalled, and so on.

However, any decent sized deployment will need to have an internally modular structure in order to fight technical debt, bloat, fragility and so on. OSGi is still in my opinion the best technology for this. JPMS is another technology that *can* be used for this... I believe it is inferior but we don't need to have that discussion here and now.

The complexity you are experiencing is because you are trying to deploy an isolated unit (the subsystem) into what is effectively an application server. Because that application server is implemented in OSGi and your deployments are also implemented in OSGi, you are pushed towards using subsystems in order to isolate the internals of your deployment from the environment you're deploying into. But it's painful because subsystems offer fairly leaky isolation, and also because they are so rarely used there is almost documentation available or community to help you. If you really want an application server you would probably be better off deploying into something much more mainstream such as J2EE, but IMHO application servers are a dead end anyway.

Regards,

Neil

To view this discussion on the web visit https://groups.google.com/d/msgid/bndtools-users/41BF86BA-97A7-4B77-8082-C7AE8ADB8202%40gmail.com.