[Hazelcast 3.2] ClassNotFound Exception after upgrading to Hazelcast 3.2

[Puneet Arora]

Hi All,

I just upgraded my application to use Hazelcast 3.2. Earlier i was using 3.0.

I made all the necessary change to the Hazelcast.xml, but Hazelcast is unable to find all the StoreLoad classes and giving the following exception.

2014-04-03 13:02:59,871 INFO  [play-thread-1] (Bootstrap.java:624) --- Exception in Bootstrap:java.lang.ClassNotFoundException: models.predmkts.StoreLoadUserMap

com.hazelcast.core.HazelcastException: java.lang.ClassNotFoundException: models.predmkts.StoreLoadUserMap

at com.hazelcast.util.ExceptionUtil.rethrow(ExceptionUtil.java:45)

at com.hazelcast.map.MapContainer.<init>(MapContainer.java:115)

at com.hazelcast.map.MapService$3.createNew(MapService.java:191)

at com.hazelcast.map.MapService$3.createNew(MapService.java:189)

at com.hazelcast.util.ConcurrencyUtil.getOrPutSynchronized(ConcurrencyUtil.java:39)

at com.hazelcast.map.MapService.getMapContainer(MapService.java:316)

at com.hazelcast.map.proxy.MapProxySupport.<init>(MapProxySupport.java:68)

at com.hazelcast.map.proxy.MapProxyImpl.<init>(MapProxyImpl.java:42)

at com.hazelcast.map.MapService.createDistributedObject(MapService.java:550)

at com.hazelcast.map.MapService.createDistributedObject(MapService.java:75)

at com.hazelcast.spi.impl.ProxyServiceImpl$ProxyRegistry.createProxy(ProxyServiceImpl.java:276)

at com.hazelcast.spi.impl.ProxyServiceImpl$ProxyRegistry.getOrCreateProxy(ProxyServiceImpl.java:254)

at com.hazelcast.spi.impl.ProxyServiceImpl.getDistributedObject(ProxyServiceImpl.java:117)

at com.hazelcast.instance.HazelcastInstanceImpl.getDistributedObject(HazelcastInstanceImpl.java:307)

at com.hazelcast.instance.HazelcastInstanceImpl.getMap(HazelcastInstanceImpl.java:135)

at com.hazelcast.instance.HazelcastInstanceProxy.getMap(HazelcastInstanceProxy.java:50)

at Bootstrap.doJob(Bootstrap.java:121)

at play.jobs.Job.doJobWithResult(Job.java:50)

at play.jobs.Job.call(Job.java:146)

at play.jobs.Job.run(Job.java:132)

at play.jobs.JobsPlugin.afterApplicationStart(JobsPlugin.java:116)

at play.plugins.PluginCollection.afterApplicationStart(PluginCollection.java:531)

at play.Play.start(Play.java:547)

at play.Play.detectChanges(Play.java:630)

at play.Invoker$Invocation.init(Invoker.java:198)

at play.server.PlayHandler$NettyInvocation.init(PlayHandler.java:189)

at play.Invoker$Invocation.run(Invoker.java:276)

at play.server.PlayHandler$NettyInvocation.run(PlayHandler.java:229)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)

at java.util.concurrent.FutureTask.run(FutureTask.java:166)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:724)

Caused by: java.lang.ClassNotFoundException: models.predmkts.StoreLoadUserMap

at java.net.URLClassLoader$1.run(URLClassLoader.java:366)

at java.net.URLClassLoader$1.run(URLClassLoader.java:355)

at java.security.AccessController.doPrivileged(Native Method)

at java.net.URLClassLoader.findClass(URLClassLoader.java:354)

at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)

at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

at com.hazelcast.nio.ClassLoaderUtil.loadClass(ClassLoaderUtil.java:109)

at com.hazelcast.nio.ClassLoaderUtil.newInstance(ClassLoaderUtil.java:62)

at com.hazelcast.map.MapContainer.<init>(MapContainer.java:112)

... 34 more

Hazelcast.xml Config file :

<map name="userMap">

<backup-count>1</backup-count>

<eviction-policy>LRU</eviction-policy>

<max-size policy="PER_NODE">0</max-size>

<eviction-percentage>25</eviction-percentage>

<merge-policy>com.hazelcast.map.merge.LatestUpdateMapMergePolicy</merge-policy>

<map-store enabled="true">

<class-name>models.predmkts.StoreLoadUserMap</class-name>

<write-delay-seconds>0</write-delay-seconds>

</map-store>

<entry-listeners>

<entry-listener include-value="true" local="false">models.predmkts.ListenerUserMap

</entry-listener>

</entry-listeners>

</map>

Kindly, inform if I am missing something. I am unable to figure out the issue.

Thanks

Puneet Arora

[Peter Veentjer]

The file can't be found on your classpath. Are you sure your application includes that particular file?

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.
To post to this group, send email to haze...@googlegroups.com.
Visit this group at http://groups.google.com/group/hazelcast.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/8c3c5bf3-ec9f-4066-9b4a-7ff5e426734c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Puneet Arora]

Yes the application includes the file. Moreover If I switch back to hazelcast-all-3.0.jar it just works out of the box.

Is there any change made to the hazelcast.xml file from 3.0 to 3.2 which may cause this issue ? Maybe I am missing something there. Just unable to figure it out.

[Puneet Arora]

Hi Peter,

The application is getting compiled, which tells me it is present in the classpath.

But it is unable to find this class when the first time I do a getMap();

Thanks

[Noctarius]

Is it possible that there are some class loaders involved when using Play?

Can you try to explicitly set the class loader in the Config before creating the HazelcastInstance.

Thanks,

Chri

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/1a84c3ef-18ec-4346-b28c-97c2d9287dc3%40googlegroups.com.

[Puneet Arora]

Hi Chri,

i tried doing that, and had a new exception generated.

Should I try it some other way ?

*************** I tried something like this ****************

Config config = new XmlConfigBuilder("conf/hazelcast.xml").build().setClassLoader(Play.classloader) ;

hcast = Hazelcast.newHazelcastInstance(config); 

**************** Exception *******************

Oops: ExceptionInInitializerError

An unexpected error occured caused by exception ExceptionInInitializerError: null

play.exceptions.UnexpectedException: Unexpected Error

at play.jobs.JobsPlugin.afterApplicationStart(JobsPlugin.java:131)

at play.plugins.PluginCollection.afterApplicationStart(PluginCollection.java:531)

at play.Play.start(Play.java:547)

at play.Play.detectChanges(Play.java:642)

at play.Invoker$Invocation.init(Invoker.java:198)

at Invocation.HTTP Request(Play!)

Caused by: java.lang.ExceptionInInitializerError

at java.lang.Class.newInstance(Class.java:374)

at play.jobs.JobsPlugin.afterApplicationStart(JobsPlugin.java:114)

... 5 more

Caused by: java.lang.IllegalArgumentException: PortableFactory[-14] is already registered! com.hazelcast.concurrent.countdownlatch.CountDownLatchPortableHook$1@7944cdbd -> com.hazelcast.concurrent.countdownlatch.client.CountDownLatchPortableHook$1@141f717e

at com.hazelcast.nio.serialization.PortableHookLoader.register(PortableHookLoader.java:84)

at com.hazelcast.nio.serialization.PortableHookLoader.load(PortableHookLoader.java:51)

at com.hazelcast.nio.serialization.PortableHookLoader.<init>(PortableHookLoader.java:41)

at com.hazelcast.nio.serialization.SerializationServiceImpl.<init>(SerializationServiceImpl.java:85)

at com.hazelcast.nio.serialization.SerializationServiceBuilder.build(SerializationServiceBuilder.java:174)

at com.hazelcast.instance.Node.<init>(Node.java:130)

at com.hazelcast.instance.HazelcastInstanceImpl.<init>(HazelcastInstanceImpl.java:95)

at com.hazelcast.instance.HazelcastInstanceFactory.constructHazelcastInstance(HazelcastInstanceFactory.java:144)

at com.hazelcast.instance.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:126)

at com.hazelcast.instance.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:102)

at com.hazelcast.core.Hazelcast.newHazelcastInstance(Hazelcast.java:58)

at Bootstrap.<clinit>(Bootstrap.java:87)

... 7 more

--

Puneet Arora, Lead Developer

______________________________________________________________

Insticator featured as the top Super Bowl companion app! - CNET
Insticator is the #1 gamification tool the competition has missed! - Business 2 Community

Recent press:   

Let's get social:    

--
You received this message because you are subscribed to a topic in the Google Groups "Hazelcast" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/hazelcast/cyIvnLRTepM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to hazelcast+...@googlegroups.com.

To post to this group, send email to haze...@googlegroups.com.
Visit this group at http://groups.google.com/group/hazelcast.

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/A45012ED-B97E-4B54-B916-D98751AD9910%40googlemail.com.

[Noctarius]

Don’t use the Play class loader, try to use one of your model classes to retrieve the class loader.

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/CAM2z126SEc5NmsJASFv6G_6Btb-PtnCiy-%2BSDTr-Fap2FO1Dqw%40mail.gmail.com.

[Puneet Arora]

I tried to use this :

config = new XmlConfigBuilder("conf/hazelcast.xml").build().setClassLoader(User.class.getClassLoader()) ;

But, same result. 

--

Puneet Arora, Lead Developer

______________________________________________________________

Insticator featured as the top Super Bowl companion app! - CNET
Insticator is the #1 gamification tool the competition has missed! - Business 2 Community

Recent press:   

Let's get social:    

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/3604C1DA-B539-402E-BF03-37B0EB641252%40googlemail.com.

[Noctarius]

Same result as for the first or the second try? ClassNotFoundException or id unknown?

Never worked with play framework on my own, is it using OSGi or something like this?

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/CAM2z124YTj4WTVyZO3XPTBoCkmWQNqRUJbLoqpMaT0zx%3D9equA%40mail.gmail.com.

[Puneet Arora]

It is the same result as second try.

Caused by: java.lang.IllegalArgumentException: PortableFactory[-14] is already registered! com.hazelcast.concurrent.countdownlatch.CountDownLatchPortableHook$1@7944cdbd -> com.hazelcast.concurrent.countdownlatch.client.CountDownLatchPortableHook$1@141f717e

 I actually am not sure that play uses OSGi or not. But doing a google search seems like they don't use it.

Thanks

--

Puneet Arora, Lead Developer

______________________________________________________________

Insticator featured as the top Super Bowl companion app! - CNET
Insticator is the #1 gamification tool the competition has missed! - Business 2 Community

Recent press:   

Let's get social:    

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/E42E8053-10F5-498A-94BA-0F4AC242500A%40googlemail.com.

[Noctarius]

That is weird, do you have the chance to create a small play app that has the same problem so I can debug it? Not sure of the classloader hierarchy play uses but it seems to do something weird. I changed a bit of the class loading mechanism from 3.1x to 3.2 to better support OSGi, so possibly I broke play for whatever reason.

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/CAM2z125d_wY4%2BncqTUop%2BLPhngT3BLwoLUTkJOAfSvymOXuxkQ%40mail.gmail.com.

[Puneet Arora]

Ok, let me create a small play application and then try it. Will get back to you in an hour. Does that work ?

--

Puneet Arora, Lead Developer

______________________________________________________________

Insticator featured as the top Super Bowl companion app! - CNET
Insticator is the #1 gamification tool the competition has missed! - Business 2 Community

Recent press:   

Let's get social:    

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/026E52D1-8C5D-40CC-B8EF-BE658F9E7662%40googlemail.com.

[Noctarius]

Take your time, it is 11pm here (Germany) so I will have a look tomorrow, will try to look at it as quickly as possible.

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/CAM2z125UVSFVeJSWpVa0SRzGc-ND%3D5kSOiX-%3D50%2Bk55hTuR%3DfQ%40mail.gmail.com.

[Puneet Arora]

Hi Noctarius,

Apologies for a very late reply, just got busy with deployment process.

Attached is a small dummy play application, having the same issue. It is not complete but yes, it replicates the issue.

This application is build using play2.2. So you can just have a copy of play 2.2 and play run.

I am very much interested in your findings as, currently we are restricting ourselves to Hazelcast 3.1, but have a lot of use cases for new features of 3.2. 

Thanks

Puneet

...

[Puneet Arora]

Hi Noctarius,

Have you got any chance to look into the play application I provided ? 

Are we missing something in Hazelcast.xml file or it is a bug ?

Thanks

Puneet

<span style="w

...

[Noctarius]

Hi Puneet,

Damn I forgot about your demo app but I was busy working on a massive amount of class loading issues, pretty much sure there is a good chance yours is fixed too. Do you have the chance to test 3.2.1-SNAPSHOT from our snapshot repository to see if it is still happening?

Chris

[Puneet Arora]

Can you send me a link to the repository? I am unable to find the SNAPSHOT repo.

Thanks

--

Puneet Arora, Lead Developer

______________________________________________________________

Insticator featured as the top Super Bowl companion app! - CNET
Insticator is the #1 gamification tool the competition has missed! - Business 2 Community

Recent press:   

Let's get social:    

[Noctarius]

Sure :)

   <repositories>

        <repository>

            <id>snapshot-repository</id>

            <name>Maven2 Snapshot Repository</name>

            <url>https://oss.sonatype.org/content/repositories/snapshots</url>

        </repository>

    </repositories>

[Puneet Arora]

Hi Chris,

We do not use maven, but I got a jar of "hazelcast-3.2.1-20140415.121247-50.jar" from typesafe.

I tried using this. But it still has the same issue. 

Let me know if I should have used another JAR file. It would be great if you can send me the JAR file directly.

Thanks

--

Puneet Arora, Lead Developer

______________________________________________________________

Insticator featured as the top Super Bowl companion app! - CNET
Insticator is the #1 gamification tool the competition has missed! - Business 2 Community

Recent press:   

Let's get social:    

[Noctarius]

Hi Puneet,

What issue? IllegalArgumentException or ClassNotFoundException?

I tried to start your play application but it seems to require a database ;-)

Some question:

- Do you use Hazelcast client or an embedded Hazelcast node?

- If client do you connect to a remote standalone cluster?

Chris

[Noctarius]

In the Hazelcast configuration conf/hazelcast.xml you define the http port as cluster port?

<tcp-ip enabled="true">

<interface>127.0.0.1:9000</interface>

</tcp-ip>

[Noctarius]

You use old config style merge-policies, for new versions you have to use full canonical classnames. Please find more information here: http://hazelcast.org/docs/latest/manual/html-single/hazelcast-documentation.html#network-partitioning-split-brain-syndrome

[Noctarius]

Ok have your application running and found the “bug”. The problem is - yeah it was changed - to fix OSGi class loading …. I hate that everybody needs to have a different class loading way -.-

[Noctarius]

Sorry it was my mistake, forgot to reset a variable before using it again. Fix is ready and PR is in workflow:

https://github.com/hazelcast/hazelcast/pull/2307/files

[Puneet Arora]

Hi Chris,

Thanks for the Bug fix. You saved me from typing a long email answering your above questions :)

Currently, there is no standalone cluster. the <tcp-ip-enabled> should be false.

We should use the new config style, actually most of it was just copy paste from an existing application which runs Hazelcast 3.0.

Thanks

--

Puneet Arora, Lead Developer

______________________________________________________________

Insticator featured as the top Super Bowl companion app! - CNET
Insticator is the #1 gamification tool the competition has missed! - Business 2 Community

Recent press:   

Let's get social:    

[Noctarius]

Important is only changing the merge-policy since those will not work for you and you’re on your way to the next classnotfound ;)