MapReduce task does not return error when mapper class not present on server (ClassNotFoundException)

[Rob Barnes]

Hi

We are running hazelcast 3.2.1 in server mode, occasionally submitting MapReduce jobs via various clients.

In the case where the mapper is not available on the server (IE jar not present), on job submission the server will spit out a ClassNotFoundException, however this is never propogated back to the client.  The client receives a Future with a job ID, with no indication the job cannot run.  When future::get() is called, obviously this then blocks forever.

I have also tried this against 3.3.

Is there any way on job submission I can get this exception and therefore not call get()?

A better solution would be to not require the jar on the server, IE serialise the class def along with the data, though not sure how technically possible this is.  I haven't investigated.

Thanks

(Package names changed to protect the innocent!)

2014/09/08 12:14 27 [FATAL] ?.?:? - [169.94.140.160]:5900 [dev] [3.2.1] java.lang.ClassNotFoundException: somepackage.SomeClass
com.hazelcast.nio.serialization.HazelcastSerializationException: java.lang.ClassNotFoundException: somepackage.SomeClass
	at com.hazelcast.nio.serialization.DefaultSerializers$ObjectSerializer.read(DefaultSerializers.java:190)
	at com.hazelcast.nio.serialization.StreamSerializerAdapter.read(StreamSerializerAdapter.java:40)
	at com.hazelcast.nio.serialization.SerializationServiceImpl.readObject(SerializationServiceImpl.java:279)
	at com.hazelcast.nio.serialization.ByteArrayObjectDataInput.readObject(ByteArrayObjectDataInput.java:433)
	at com.hazelcast.mapreduce.impl.client.ClientMapReduceRequest.readData(ClientMapReduceRequest.java:220)
	at com.hazelcast.mapreduce.impl.client.ClientMapReduceRequest.read(ClientMapReduceRequest.java:175)
	at com.hazelcast.client.ClientRequest.readPortable(ClientRequest.java:101)
	at com.hazelcast.nio.serialization.PortableSerializer.read(PortableSerializer.java:99)
	at com.hazelcast.nio.serialization.PortableSerializer.read(PortableSerializer.java:29)
	at com.hazelcast.nio.serialization.StreamSerializerAdapter.read(StreamSerializerAdapter.java:59)
	at com.hazelcast.nio.serialization.SerializationServiceImpl.toObject(SerializationServiceImpl.java:221)
	at com.hazelcast.client.ClientEngineImpl$ClientPacketProcessor.loadRequest(ClientEngineImpl.java:457)
	at com.hazelcast.client.ClientEngineImpl$ClientPacketProcessor.run(ClientEngineImpl.java:433)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
	at java.lang.Thread.run(Thread.java:722)
	at com.hazelcast.util.executor.PoolExecutorThreadFactory$ManagedThread.run(PoolExecutorThreadFactory.java:59)
Caused by: java.lang.ClassNotFoundException: somepackage.SomeClass
	at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
	at com.hazelcast.nio.ClassLoaderUtil.loadClass(ClassLoaderUtil.java:113)
	at com.hazelcast.nio.IOUtil$1.resolveClass(IOUtil.java:89)
	at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1593)
	at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1514)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1750)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
	at com.hazelcast.nio.serialization.DefaultSerializers$ObjectSerializer.read(DefaultSerializers.java:185)
	... 16 more

[Noctarius]

Hi Rob,

Have you tried it with 3.3 GA? Exactly that should be fixed a few days ago where the client wasn't able to get exceptions propagated in those cases.

Chris

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.
To post to this group, send email to haze...@googlegroups.com.
Visit this group at http://groups.google.com/group/hazelcast.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/16d7b218-efb0-4907-8b40-f87306d7cf76%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Serkan Özal]

Hi Rob,

For the class definition serialization, it is not seem feasible since they can be dependent to other classes and they must be handled for serialization and also their dependency graph. So I think it is not simple as just serialize the actual class.

[Rob Barnes]

I tried this against the 3.3 version released yesterday, I still get the same issue. :(

3.3 seems to have made security an enterprise feature also, which may prohibit upgrading even if it were fixed :(

[Noctarius]

Oh I see I fixed it between nodes but I missed out the client requests, sh.... :(
What kind of security features do you need?

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/7e2159e4-60e6-4bc1-bda8-f1483eb1efa3%40googlegroups.com.

[Serkan Özal]

Another solution may be using Hazelcast cluster aware URLClassLoader for application classloader.

 

At first, requested class is searched at current JVM. 

If cannot find, class-data can be searched at global class-data map on Hazelcast cluster. 

If not found gain, class-data can be requested from nodes in cluster and put to global class-data map on Hazelcast cluster. 

[Rob Barnes]

Is this actually a current feature the URLClassLoader for 3.2.1?  Our mappers are quite simple typically using only standard java types.

If so I can investigate this as a solution.

[Rob Barnes]

Ahh, well that would explain it, so no fix currently!

Security wise we were planning to implement some kind of security to ensure all clients and nodes cannot access the data without some kind of secret.  Symmetric would have sufficed but looks to have been taken out at 3.3 to be made an enterprise feature.

Worst case we can encrypt the data itself at the data layer ensuring all clients need the shared key to decrypt, just surprised some features are actually being removed from the open source edition.

[Serkan Özal]

Not a feature currently for Hazelcast and it just my personal idea and not sure it is correct or not :) 

And all your contributions are welcome for us :) and you can send us a pull request about it. And I think, it will be great contribution.

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.
To post to this group, send email to haze...@googlegroups.com.
Visit this group at http://groups.google.com/group/hazelcast.

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/559b7177-c3ac-4589-b17e-fd6b90df729a%40googlegroups.com.

[Noctarius]

Hi Rob,

I created an issue on Github and fixed the bug for 3.3.1 if this helps you:
https://github.com/hazelcast/hazelcast/issues/3557

Chris

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/CA%2BMwa3OcbkyroP2TRgzn6%2BaTe63Ci9-VraxgFS9nS1sax8-v7A%40mail.gmail.com.