Using Libraries in JUnit tests

[i...@openfokus.org]

I have (re)-read :

1) http://enroute.osgi.org/doc/215-sos.html:

Libraries

A library is a bundle without internal state and should therefore have no components or Bundle Activator. A good library provides an API to do certain tasks but all observable state is maintained in instance variables and never in statics. Libraries should be started like all bundles but will not act differently whether they are started or not because they lack state. Libraries never register or use services.

The purpose of a library is to provide convenience functions for a specific area. Libraries do not provide substitution.

2) http://enroute.osgi.org/doc/300-principles.html:

The second technique to reduce dependency hell is to separate an Application Programmer Interfaces (API) from the implementation, even if there will only be a single implementation for this API ever.

3) http://enroute.osgi.org/tutorial_wrap/212-conditional-package:

The bnd tool contains a nifty feature that it inherited from old school C. In those days, the link phase searched for missing symbols in libraries and automatically included those functions from the library in your code. This was called static linking. Neil Bartlett wrote a blog about the subject.

The -conditionalpackage instruction performs a similar function for packages.

All this material, and in particular the detailed wrapper-example in 3) is incredibly useful to me, many thanks !

My problem is : I have wrapped a couple of non-osgi libraries as described in 3) that do not really have an api and export a huge number of types and methods. I have build a wrapper bundle that contains all dependencies as shown in 3) but
it still has no proper api. I would like to create an api project and a provider project as described in the tutorials because I only need some types and some methods from these libs and I understand the merits of 2).
But I also need these libraries during unit tests ( that is, not only in osgi tests ). So they must be usable without running inside a framework. For example, I use a parser, a runtime compiler and a code formatter as part of unit tests.

Then, the first point 1) states that a library should have no components or BundleActivator. Libraries should be started like all bundles ...

What does this mean ? How can they be started like all bundles without having an activator ( or being a component ) ?

My question is: what is the best way to achieve this double goal of a) reducing deps as suggested in 2) and building libraries in the sense of 1) that are not components and can therefore be used "everywhere" , including in unit test
(i.e do not require to run in a framework). That would be no problem of course in standard java.  I think I don't really understand the notion of library in the context of OSGi.

Many thanks for your suggestions
Peter

[Peter Kriens]

I would say do not try to boil the ocean …

In the most cases when a non-OSGi library is useful I find that a lot of functionality overlaps with the OSGi functionality. E.g. most libraries have their own configuration model and factory models. (This is why OSGi bundles can be so much smaller than comparable non-OSGi JARs.) You therefore do not want to drag in this overlapping functionality in the public space. So in general my strategy is to reverse the perspective. Instead of thinking how I can provide the functionality of library X into OSGi I look at what I need from library X and how can I represent that as a good service. So I am not trying to turn Hibernate or Apache Shiro in a good OSGi citizen (you can’t) but I abstract the functionality that I need from that library and wrap it in a service that only does what I need it to do in the way that I need it done. For example, I can implement the OSGi enRoute Authenticator on Shiro without even exposing a single class of this library. I then wrap this pure OSGi service code in a bundle that hides the library in the private area of a bundle so it won’t conflict with anybody. I the use Configuration Admin to configure my functionality. So it clearly is not as flexible and widely applicable as the original but its API surface has dramatically shrunk. And the OSGi iceberg model (the top of the iceberg is the service, the implementations are invisible) is what makes OSGi work so well in larger applications.

The more cohesive and 1 dimensional you make the API, the easier is it to create a mock for JUnit testing. In a project I work we’ve started to standard create ‘Dummy’ classes that are very easy to use in testing but do not drag in other dependencies. The better more cohesive you API, the easier this is. For example, you can write a Configuration Admin dummy class in less than an hour. (OSGi should provide those dummy classes for all OSGi services!). This approach is really hard in most non-OSGi software because there are so many dependencies but it is enabled by OSGi’s service model when properly used.

In my long experience it is often surprisingly little work to create a good service API on a huge library if you take the selfish perspective of what YOU need, not what the library could do for you. If you design the service to not use all kind of internal types but make it as abstract as your problem allows then you find that you even could switch to a different library in the future.

In Service Oriented Design the all overriding issue is to place functionality in the right bundle so that the services they provide are as cohesive and 1 dimensional as possible.  Such services are easy to combine and easy to maintain into applications because they allow you to make huge changes on implementation level.

Sometimes this means the same library is present in multiple bundles. Well, that is bad but much better than the alternative of exposing the often huge API and transitive dependencies to the internal OSGi bundles.

One thing I often discover when doing this approach is that the problem I need to solve translates in a very simple service API and I do not need this big fat library at all. :-)

Kind regards,

Peter Kriens

--
You received this message because you are subscribed to the Google Groups "bndtools-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bndtools-user...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[i...@openfokus.org]

Thanks Peter for these thoughts and recommendations, I totally agree. But in french, the reference language, the saying is, I think " ce n'est pas la mer à boire" rather than "ne pas faire cuire la mer" (:,

But I think I am missing something very simple. I generally do what you recommend, namely write a thin wrapper around a library, that exposes just what I need. But, this is now a bundle and must run in a framework.
How can I use this in normal unit tests. Think about a something like String.format() but more complex, comming from such a wrapped library. I would like to use this like I use the jdk.
Is this wrong ?


Peter

[Peter Kriens]

For JUnit tests, you will have two. One for the wrapper code, this tests the API and requires the same kind of testing as the library you use. The other JUnit tests are written against a dummy implementation/mock of your API. These units can, imho, be written in almost all cases without access to an OSGi framework. During integration tests your code and your wrapper meet for the first time and run on OSGi.

This is the only way to keep these two different projects separate and allow for implementation substitution in the future.

Kind regards,

Peter Kriens

[i...@openfokus.org]

So this implies, if I undertand correctly that, for the Library type of "component" , as described on the osgi enroute website,

1)  wrap the entire library as explained for Dom4j
2)  write an api project that exposes what you need from this library ( as little as possible ). compile-only
3)  write a provider project that implements this api and that has the wrapped lib from step 1) on the buildpath. The provider package exports the api from step 2) and nothing else.
4) if this library is also useful to test implementation code of other bundles , that is as part of junit code, write a mock implementation for the api in step 2) so that it can be used outside a framework


Hmm, I think I still don't get it. In step 4 , I need a _production_ version of the wrapped lib, not a mock. Its like using something from the jdk, or apache commons or whatever in the traditional way. Only
that I want to use my osgi-wrapped version with its tightly controlled api.

Peter

[Peter Kriens]

On 6 Mar 2017, at 10:32, i...@openfokus.org wrote:

So this implies, if I undertand correctly that, for the Library type of "component" , as described on the osgi enroute website,

1)  wrap the entire library as explained for Dom4j
2)  write an api project that exposes what you need from this library ( as little as possible ). compile-only 
3)  write a provider project that implements this api and that has the wrapped lib from step 1) on the buildpath. The provider package exports the api from step 2) and nothing else.
4) if this library is also useful to test implementation code of other bundles , that is as part of junit code, write a mock implementation for the api in step 2) so that it can be used outside a framework 

Hmm, I think I still don't get it. In step 4 , I need a _production_ version of the wrapped lib, not a mock. Its like using something from the jdk, or apache commons or whatever in the traditional way. Only that I want to use my osgi-wrapped version with its tightly controlled api.

Well, if you want that why not put the implementation bundle on the classpath? The compiler + junit will be able to see your private packages. So you have no modularity but that is the price to pay if your mock is not sufficiently good enough. Private package access is indicated by yellow colored squigles under the access but you can get rid of them by adding ‘packages=*’ on the buildpath where you add the library.

Kind regards,

Peter Kriens

[i...@openfokus.org]

What about  using an embedded framework and put the library there.

class JunitTest {

  private Framework fw =  ....

  ILibraryApi lib =  fw.getService(ILibraryApi.class);

   @Test public void test() {
        lib.function() ;
       .....

Is this overkill ? I would get complete encapsulation and the library can be used either inside a framework or outside for unit tests or command line apps.

There are some details to figure out  to populate the framework with the required bundles ( from bnd.bnd.resolved ) and fill the config map with the system packages.

I have already all system related packages in a ext.bnd ( -runsystempackages)   in ./cnf and the user packages are in the manifest. (Export-Package)

This would be a kind of plug-in system based on OSGi.

Nonsense ?

[Peter Kriens]

Yes, it works but there are some nasty caveats. :-(

The problem is that the framework must export ANY package that your test code relies on, and in the proper version(s). This is really hard to do manually. Included you find a class I intend to add to bnd once I can use it again after the enRoute templates have been restored that were removed.

The DummyFramework class uses the bnd information to create a virtual bundle of the test classes and then uses the imports and contained packages to create the Framework extra exported packages. The other class demoes some of the usage that the framework provides since it can also create and install bundles on the fly. As a dependency you need to add biz.aQute.bnd or in maven GAV biz.aQute.bnd:biz.aQute.bnd:3.3.0.

This is work in progress but it seems to actually work quite nicely. 

Kind regards,

Peter Kriens

[i...@openfokus.org]

Peter

I posted a rather long reply to your mail and I am sure the post got sent. But now , about an hour later I see:

This message has been deleted.

instead of my reply. No idea what that means. Did you see my reply ?

Peter

[Peter Kriens]

Not seen it …

[i...@openfokus.org]

Hm, that's weird, are there restrictions on what can be posted ( hyperlinks, source code ) ?.

Anyway, many thanks for the code example. When I run it on my test bundle, I get:

extras:
1) org.jdom2,org.jdom2.input,org.jdom2.output,

2) softbrix.lib.type.common;version=1.0.0,
   softbrix.lib.type.fs;version=1.0.0,
     softbrix.lib.h;version=1.0.0,

3) softbrix.xml.lib.api;version=null,

4) softbrix.xml.lib.impl


1) (org.jdom2) is the external lib I want to wrap and it is referenced only in the private package section.
It should , I think, not be part of the extras, because it will never be used from outside of the bundle.

2) softbrix contains classes that I use to wrap other libraries. Has no dependencies.

3) this is the api of the new library that wraps 1). The "api" package is in the export package section.

4) this is the impl of the new library and is in the private package section. It should not be part of the extras.


So it seems that this extra package list is not exactly what I want. Only 2) and 3) should be part of it.

In the sample you sent, you use the "Import" section ( in getExtras()) .

In my current test implementation, I use the export section.

For the above bundle this is:

Export-Package: softbrix.xml.lib.api;version="1.0.0";
        uses:="softbrix.lib.type.common,softbrix.lib.type.fs,softbrix.lib.type.io,softbrix.lib.type.list,softbrix.lib.type.result"
       
From this I computed the extras:
softbrix.xml.lib.api;version="1.0.0", softbrix.lib.type.common;version="1.0.0", softbrix.lib.type.fs;version="1.0.0", softbrix.lib.type.io;version="1.0.0", softbrix.lib.type.list;version="1.0.0", softbrix.lib.type.result;version="1.0.0"

This works sometimes. The problem seems to be that if I instantiate a type
in the private section, that is not part of the export section, I get errors like these :

java.lang.LinkageError: loader constraint violation:
when resolving method "softbrix.lib.fpj.result.R.map(Lsoftbrix/lib/fpj/common/F;)Lsoftbrix/lib/fpj/result/R;"
the class loader (instance of org/apache/felix/framework/BundleWiringImpl$BundleClassLoaderJava5) of the current class,
robobrix/template/lib/impl/SimpleTemplate,
and the class loader (instance of sun/misc/Launcher$AppClassLoader) for the method's defining class,

While I think that I understand what the error means, I don't know if there exists a necessary and sufficient condition that garantees that this error
never occurs - other than using the import super set which seems to be too large.

I can then fix this error by manually adding the missing types to the exports. But this is not satisfying because I only
discover this at runtime.


So as it stands, it seems to me, that using the imports - as you do in your sample - results in too many extras, but is this a problem ?
On the other hand, relying on the exports may result in LinkageErrors which is certainly a problem.

So using the imports is the way to go ?

[Peter Kriens]

It is work in progress. That said.

On 8 Mar 2017, at 14:20, i...@openfokus.org wrote:

Hm, that's weird, are there restrictions on what can be posted ( hyperlinks, source code ) ?.
Anyway, many thanks for the code example. When I run it on my test bundle, I get:

extras: 
1) org.jdom2,org.jdom2.input,org.jdom2.output,

2) softbrix.lib.type.common;version=1.0.0,
   softbrix.lib.type.fs;version=1.0.0,
     softbrix.lib.h;version=1.0.0,

3) softbrix.xml.lib.api;version=null,

4) softbrix.xml.lib.impl


1) (org.jdom2) is the external lib I want to wrap and it is referenced only in the private package section.
It should , I think, not be part of the extras, because it will never be used from outside of the bundle.

2) softbrix contains classes that I use to wrap other libraries. Has no dependencies.

3) this is the api of the new library that wraps 1). The "api" package is in the export package section.

4) this is the impl of the new library and is in the private package section. It should not be part of the extras.

It should also cause no harm … If you really want to prevent this then you have to move your test to another program or subclass DummyFramework and override getExtra()

So it seems that this extra package list is not exactly what I want. Only 2) and 3) should be part of it.

I disagree … when you unit test you have full visibility of the projects buildpath + testpath. So in the DummyFramework I expose these classes. If you control that then you will go bananas quickly. 

You cannot have it both ways. Or you make proper OSGi integration tests giving you full protection or you do unit tests where you use scaffolding with a small change you fall from the scaffold now and then.

In the sample you sent, you use the "Import" section ( in getExtras()) .

In my current test implementation, I use the export section.

For the above bundle this is:

Export-Package: softbrix.xml.lib.api;version="1.0.0"; 
        uses:=“softbrix.lib.type.common,softbrix.lib.type.fs,softbrix.lib.type.io,softbrix.lib.type.list,softbrix.lib.type.result"

This will not work for normal JUnit tests because there are always lots of transient dependencies to the build- and testpath.

From this I computed the extras:
softbrix.xml.lib.api;version="1.0.0", softbrix.lib.type.common;version="1.0.0", softbrix.lib.type.fs;version="1.0.0", softbrix.lib.type.io;version="1.0.0", softbrix.lib.type.list;version="1.0.0", softbrix.lib.type.result;version="1.0.0"

This works sometimes. The problem seems to be that if I instantiate a type 
in the private section, that is not part of the export section, I get errors like these :

java.lang.LinkageError: loader constraint violation: 
when resolving method "softbrix.lib.fpj.result.R.map(Lsoftbrix/lib/fpj/common/F;)Lsoftbrix/lib/fpj/result/R;" 
the class loader (instance of org/apache/felix/framework/BundleWiringImpl$BundleClassLoaderJava5) of the current class, 
robobrix/template/lib/impl/SimpleTemplate,
and the class loader (instance of sun/misc/Launcher$AppClassLoader) for the method’s defining class,

Yup. That is the kind of shit you get because now the inside OSGi framework is going to get confused where classes come from.

While I think that I understand what the error means, I don't know if there exists a necessary and sufficient condition that garantees that this error
never occurs - other than using the import super set which seems to be too large.

Why is it too large? 

I can then fix this error by manually adding the missing types to the exports. But this is not satisfying because I only
discover this at runtime.

I really do not understand why you get upset about the extras. They should in general be harmless for unit tests. And if you make a mistake you catch those the first time you do an integration tests because that would be the stuff that OSGi is really good in detecting ahead of time.

So as it stands, it seems to me, that using the imports - as you do in your sample - results in too many extras, but is this a problem ?
On the other hand, relying on the exports may result in LinkageErrors which is certainly a problem.

Yes, exports will never work. The extras bridge the classic classpath world with the OSGi inner framework. If you’re not taking the imports into account the JUnit code sees the classpath version and the inner framework classes see the bundle versions. Receipt for a disaster.

So using the imports is the way to go ? 

I do not think there is an alternative …

Kind regards,

Peter Kriens

[i...@openfokus.org]

Ok, many thanks, I think I got it. I'll continue to work based on the code you posted.

I try to familarize myself with the bnd java api, in particular the Analyzer class.

Peter

[Peter Kriens]

On 9 Mar 2017, at 11:00, i...@openfokus.org wrote:

Ok, many thanks, I think I got it. I'll continue to work based on the code you posted.
I try to familarize myself with the bnd java api, in particular the Analyzer class.

Don’t overdo it. Clearly I think bnd is great but you do not in general want to couple your build on that level to bnd unless you have severe problems.

Kind regards,

Peter Kriens

[peter....@openfokus.org]

I have put toegether a repo github.com:vittali/embedded-library.git .

I have also created an issue where I need help ...

Once the basic mechanism works, I would like to add more realistic examples.

Peter