classloading issues / how to use setClassLoader()

[Christian Migowski]

Hi,

we're experiencing classloading issues with Hazelcast. A little bit of
background: We are using a managed environment which loads each
component with different classloaders (Apache Servicemix, service
assemblies are loaded with different classloaders). We are also
starting a custom JMX agent connector server which is started as a
java premain (-javaagent). When we want to access data structures
stored in Hazelcast distributed maps via JMX, we are getting
java.lang.ClassNotFoundException from AbstractSerializer because it
obviously tries to load the class with the current threads
contextclassloader, which is not working in our case.
Since we are packaging Hazelcast together with our own classes into
the service assembly which is loaded by Servicemix, for us it would be
good if the classes could be loaded by the classloader that loaded the
Hazelcast classes, not the current threads ones, but I saw this was
changed for a good reason a while back
(http://code.google.com/p/hazelcast/issues/detail?id=276).
Searching the web for a solution I found Config.setClassLoader, which
seems like could solve our problems, but unfortunately I do not find
any examples on how to use it. Could you please post a small example
on how to tell Hazelcast to use a different classloader than
Thread.currentThread().getContextClassLoader() ?

We are currently using Hazelcast very simple, just Hazelcast.getMap(),
we can certainly add code that sets the classloader in our code.

thanks and best regards,
christian!

[Talip Ozturk]

Config config = new XmlConfigBuilder().build();
config.setClassLoader(myClassLoader);
HazelcastInstance hazelcast = Hazelcast.newHazelcastInstance(config);

from now on, hazelcast instance will use your classloader.

If you are using singleton Hazelcast class then you should call
Hazelcast.init(config) first.

http://twitter.com/oztalip

> --
> You received this message because you are subscribed to the Google Groups "Hazelcast" group.
> To post to this group, send email to haze...@googlegroups.com.
> To unsubscribe from this group, send email to hazelcast+...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/hazelcast?hl=en.
>
>

[Anders Engström]

Hi!

@Talip - will that really work for loading map-entries? As far as I understand, the class-loader set through the config will only be used to load MapStore-related stuff, MergePolicies and by the ExecutorService stuff?

To unmarshal map-entries the TCCL is still used, right? This is an issue for us as well, since we run in an OSGi environment :/ 

//Anders

[Talip Ozturk]

You are right. We really do need an OSGi expert to figure these all out.

-talip

[Anders Engström]

One initial approach would be to actually allow the Config#setClassLoader to propagate even to the AbstractSerializer if set. That way it's at least possible to create a custom class-loader that will work in an OSGi scenario (using Dynamic-ImportPackage or by hacking a custom CL that uses the PackageAdmin).

//Anders

[Talip Ozturk]

2011/9/22 Anders Engström <epir...@gmail.com>:

Anders, can you send a patch?

[Anders Engström]

Sure - I will give it a try. I'll create an issue and attach the patch
to the issue instead. Will post in the thread when done.

Best Regards //Anders

[Talip Ozturk]

Sounds good!

http://twitter.com/oztalip

2011/9/22 Anders Engström <epir...@gmail.com>:

[Ioannis Canellos]

No Fuad,

No the patch I sent will not help here. The patch will just avoid "caching" of loaded classes which is dangerous for modular applications, since a class may change and having the class "cached" may result in class cast exception.

If I follow the thread correctly, Christian who started the thread is using ServiceMix 3.x or ServiceMix 4.x but with JBI packaging rather than OSGi packaging. So according to my understanding this is not a pure OSGi related issue.

It would be nice if it was possible to set the class loader inside hazelcast config and then have the AbstractSerializer reuse that class loader. Last time I checked the AbstractSerializer did not make use of that class loader.

@Andreas: Why is it an issue for you to use the TCCL inside OSGi?. Its something that works and without affecting the modularity of your applications. I agree that is some cases having the class loader being set in the configuration would help, but I don't see a problem with the TCCL inside OSGi.

--
Ioannis Canellos

FuseSource

Blog: http://iocanel.blogspot.com

Apache Karaf Committer & PMC

Apache ServiceMix  Committer

Apache Gora Committer

[Anders Engström]

On Thursday, September 22, 2011 6:07:09 PM UTC+2, Ioannis Canellos wrote:

@Andreas: Why is it an issue for you to use the TCCL inside OSGi?. Its something that works and without affecting the modularity of your applications. I agree that is some cases having the class loader being set in the configuration would help, but I don't see a problem with the TCCL inside OSGi.

Using the TCCL in OSGi is generally a bad idea. Here's some examples:

http://stackoverflow.com/questions/2198928/better-handling-of-thread-context-classloader-in-osgi

http://old.nabble.com/Can-the-thread-context-classloader-issue-be-solved-at-all--td28260809.html#a30704352

http://njbartlett.name/2010/08/30/osgi-readiness-loading-classes.html

Basically (at least in our system) the TCCL is at best "undefined". The following scenario is typical:

1. BundleA puts an entry in the hz-map: Hazelcast.getMap("foo").put("bar", reference)

2. Later on, BundleA reads the entry back: Hazelcast.getMap("foo").get("bar") (on the same JVM or on another JVM)

3. The TCCL in this case is undefined - most likely pointing to the AppClassLoader of the JVM (which does not see BundleA).

You could setup the TCCL manually before calling map.get("..") but that adds un-necessary complexity - and will not work in all scenarios (it iterating over the values() in the map outside of the calling code, for example).

I'll try to create a patch and test-case to allow the class-loader set in Config to be used by the serializer as well.

Another option (which I haven't really investigated) could be to add a BundleActivator that would add a custom branch in com.hazelcast.nio.SerializationHelper that will write some meta-data about the bundle which owns the class to the DataOutput (which can be used to find the correct bundle when reading the entry back).

//Anders

[Anders Engström]

Ok, after spending some time in the code I can't really find a clean way of providing the AbstractSerializer with a reference to the Config of the 'current node' :/ Since the serialization-stuff isn't really scoped to a specific hazelcast-instance it would require a bit of (ugly) patching and possible some thread-locals in order to find the current instance's config. Basically, in order to get the serialization scoped by hazelcast-instance a major refactoring is required (please correct me if I've missed something).

I think that from an OSGi-perspective it should be possible to create a TypeSerializer that can serialize an object together with the symbolic name of its host-bundle. That TypeSerializer would be installed/configured statically in the DefaultSerializer by a BundleActivator (this way there would be no runtime dependency on OSGi, and the custom serialization strategy would only be activated if running inside an OSGi container). 

Any thoughts on this?

//Anders

[Ioannis Canellos]

Using the TCCL in OSGi is generally a bad idea. Here's some examples:

http://stackoverflow.com/questions/2198928/better-handling-of-thread-context-classloader-in-osgi

http://old.nabble.com/Can-the-thread-context-classloader-issue-be-solved-at-all--td28260809.html#a30704352

http://njbartlett.name/2010/08/30/osgi-readiness-loading-classes.html

If you know what you are doing, using the TCCL is not a bad idea. If you see things from the framework developer perspective, a framework can load user class with the following ways:

a) Use fragment bundles (put user classes in a bundle that is going to get attach to the framework bundle).

b) Use dynamic imports.

c) Use buddy class loading

d) Use custom class loaders

e) Pass the right classloader to the framework or the TCCL.

Fragments: Will require to be installed before the framework bundle or will require refresh, which might be unwanted.

Dynamic Imports: Can result in constraint violation and other issues such as unwanted refreshes etc.

Buddy Classloading: Only available to equinox, but still compromises modularity.

Custom Classloader: By custom classloaders, I mean classloader implementation that will try to delegate classloading to the correct bundle, either by registration (an extender possibly) or by a lookup (I think you mentioned package admin). This can only be a last resort, since they will only be able to trigger their lookups using the className but the classloader api doesn't have means to pass version information.

So the last solution is to somehow pick the right class loader from the framework or from the TCCL. If you see the source of most osgi ready frameworks, you will find out that most of them solve the problem this way.

Of course nobody can disagree that this needs to be hidden from the framework user and that the user should not put his fingers on the TCCL. This is something that needs to be done by the framework internally. So the question is how Hazelcast could hide that from the user.

 

Basically (at least in our system) the TCCL is at best "undefined". The following scenario is typical:

1. BundleA puts an entry in the hz-map: Hazelcast.getMap("foo").put("bar", reference)

2. Later on, BundleA reads the entry back: Hazelcast.getMap("foo").get("bar") (on the same JVM or on another JVM)

3. The TCCL in this case is undefined - most likely pointing to the AppClassLoader of the JVM (which does not see BundleA).

You could setup the TCCL manually before calling map.get("..") but that adds un-necessary complexity - and will not work in all scenarios (it iterating over the values() in the map outside of the calling code, for example).

Agree, this is what I mean that needs to be hidden from the framework user.

 

[Christian Migowski]

Hi,

On Thu, Sep 22, 2011 at 6:07 PM, Ioannis Canellos <ioc...@gmail.com> wrote:

> If I follow the thread correctly, Christian who started the thread is using
> ServiceMix 3.x or ServiceMix 4.x but with JBI packaging rather than OSGi
> packaging. So according to my understanding this is not a pure OSGi related
> issue.

Yes, this is correct. We're using Servicemix 3.2.2 and as said the
problem arises because Hazelcast tries to load our custom classes with
the TCCL - which is in case of JMX access the "system classloader" and
this one does not know about our classes because our classes were
loaded together with Hazelcast by the Servicemix service
engine/assembly classloader.

For us the a simple solution like the one provided in the attached
patch (made from current trunk) would work, but I understood from your
discussion (which was btw. quite insightful; thanks for the links
Anders!) that this still would not work for all OSGI cases. However,
the patch makes the current situation a little bit better.

Thanks and best regards,
christian!

> It would be nice if it was possible to set the class loader inside hazelcast
> config and then have the AbstractSerializer reuse that class loader. Last
> time I checked the AbstractSerializer did not make use of that class loader.
> @Andreas: Why is it an issue for you to use the TCCL inside OSGi?. Its
> something that works and without affecting the modularity of your
> applications. I agree that is some cases having the class loader being set
> in the configuration would help, but I don't see a problem with the TCCL
> inside OSGi.
> --
> Ioannis Canellos
> FuseSource
>
> Blog: http://iocanel.blogspot.com
> Apache Karaf Committer & PMC
> Apache ServiceMix  Committer
> Apache Gora Committer
>
>
>
>
>

[Christian Migowski]

OK, after thinking about it a few minutes more the approach to "allow"
ClassNotFoundExceptions happening all the time seems not very clever,
maybe another System property could be introduced that if set to true
skips the TCCL retrival and loading alltgether

private static final boolean USELOCALCLASSLOADER =
Boolean.valueOf(System.getProperty("hazelcast.serializer.use.local.classloader",
"false"));

...
if(!USELOCALCLASSLOADER)
theClassLoader = Thread.currentThread().getContextClassLoader();

[Anders Engström]

On Thursday, September 22, 2011 9:14:19 PM UTC+2, Ioannis Canellos wrote:

Using the TCCL in OSGi is generally a bad idea. Here's some examples:

http://stackoverflow.com/questions/2198928/better-handling-of-thread-context-classloader-in-osgi

http://old.nabble.com/Can-the-thread-context-classloader-issue-be-solved-at-all--td28260809.html#a30704352

http://njbartlett.name/2010/08/30/osgi-readiness-loading-classes.html

If you know what you are doing, using the TCCL is not a bad idea. [snip]

I don't agree with you there - but maybe we're looking at the problem from different POVs. :)

 

Fragments: Will require to be installed before the framework bundle or will require refresh, which might be unwanted.

That's one approach that we've considered, but it is a bit complex, and introduces a 'circular' dependency between the application-bundle -> fragment-bundle -> hazelcast-bundle (since the application-bundle needs to import its 'map-entry classes' from the hazelcast-bundle indirectly through the fragment-bundle, which itself is really a part of the application-model). This also requires that Hazelcast uses its own class-loader (not the TCCL) to load classes.

 

Dynamic Imports: Can result in constraint violation and other issues such as unwanted refreshes etc.

I agree, this is not the best solution. We're using it with Hibernate in our system - and we're aware that we will not be able to support different bundle versions for example. But it mostly works.

 

Buddy Classloading: Only available to equinox, but still compromises modularity.

Not usable for us.

 

Custom Classloader: By custom classloaders, I mean classloader implementation that will try to delegate classloading to the correct bundle, either by registration (an extender possibly) or by a lookup (I think you mentioned package admin). This can only be a last resort, since they will only be able to trigger their lookups using the className but the classloader api doesn't have means to pass version information.

They could be configured to include version information as well, at least if we were to extend the Hazelcast serialization engine with pluggable OSGi-aware TypeSerializer. The bundle which "owns" a class can be found through org.osgi.framework.FrameworkUtil#getBundle(). This information (symbolic bundle-name and version) can be written as part of the serialized map-entry.

 

So the last solution is to somehow pick the right class loader from the framework or from the TCCL. If you see the source of most osgi ready frameworks, you will find out that most of them solve the problem this way.

I rather see a pluggable OSGi-aware serializer strategy than trying to 'fix' the TCCL in an OSGi runtime environment.

 

Of course nobody can disagree that this needs to be hidden from the framework user and that the user should not put his fingers on the TCCL. This is something that needs to be done by the framework internally. So the question is how Hazelcast could hide that from the user.

 

Basically (at least in our system) the TCCL is at best "undefined". The following scenario is typical:

1. BundleA puts an entry in the hz-map: Hazelcast.getMap("foo").put("bar", reference)

2. Later on, BundleA reads the entry back: Hazelcast.getMap("foo").get("bar") (on the same JVM or on another JVM)

3. The TCCL in this case is undefined - most likely pointing to the AppClassLoader of the JVM (which does not see BundleA).

You could setup the TCCL manually before calling map.get("..") but that adds un-necessary complexity - and will not work in all scenarios (it iterating over the values() in the map outside of the calling code, for example).

Agree, this is what I mean that needs to be hidden from the framework user.

 

The problem is that in a concurrent environment the thread that 'reads' the entry does not have to be the thread that deserializes the payload.

//Anders