Hazelcast with global serializer (Kryo) - There is no suitable de-serializer for type

[hoda...@gmail.com]

I’m using Hazelcast 3.9 to cluster user sessions.

To serialize the session objects, I created a global serializer implemented with Kryo (or more precisely KryoReflectionFactorySupport that allow to serialize objects without default constructor).

public class GlobalKryoSerializer implements StreamSerializer<Object> {
    //use ThreadLocal because Kryo is not thread safe
    private static final InheritableThreadLocal <Kryo> kryoThreadLocal = new InheritableThreadLocal <Kryo>() {
        @Override
        protected Kryo initialValue() {
           Kryo kryo = new KryoReflectionFactorySupport();
           //Kryo uses its own class loader       
           kryo.setClassLoader(java.lang.Thread.currentThread().getContextClassLoader());
           return kryo;
        }
    };

    public GlobalKryoSerializer(){    }

    public int getTypeId() {
        return 123;
    }

    public void destroy() {    }

    public void write(ObjectDataOutput objectDataOutput, Object object) throws IOException {
        Output output = new Output((OutputStream) objectDataOutput);
        Kryo kryo = kryoThreadLocal.get();
        kryo.writeClassAndObject(output, object);
        output.flush();
    }

    public Object read(ObjectDataInput objectDataInput) throws IOException {
        InputStream in = (InputStream) objectDataInput;
        Input input = new Input(in);
        Kryo kryo = kryoThreadLocal.get();
        Object retVal = kryo.readClassAndObject(input);
        return retVal;
    }
}

Hazelcast.xml is similar to https://github.com/hazelcast/hazelcast/blob/master/hazelcast/src/main/resources/hazelcast-default.xml with global serializer configuration:

<serialization>
      <portable-version>0</portable-version>
      <serializers>
           <global-serializer override-java-serialization="true"> GlobalKryoSerializer</global-serializer>       
      </serializers>
</serialization>

When the session is invalidated, Hazelcast fails to de-serialize with the error below (the “type number” varies from time to time):

com.hazelcast.nio.serialization.HazelcastSerializationException: There is no suitable de-serializer for type 16843028. This exception is likely to be caused by differences in the serialization configuration betw een members or between clients and members. at com.hazelcast.internal.serialization.impl.AbstractSerializationService.newHazelcastSerializationException(AbstractSerializationService.java:236) ~[hazelcast-all-3.9.jar:3.9] at com.hazelcast.internal.serialization.impl.AbstractSerializationService.readObject(AbstractSerializationService.java:263) ~[hazelcast-all-3.9.jar:3.9] at com.hazelcast.internal.serialization.impl.ByteArrayObjectDataInput.readObject(ByteArrayObjectDataInput.java:570) ~[hazelcast-all-3.9.jar:3.9] at com.hazelcast.session.HazelcastSession.deserializeMap(HazelcastSession.java:141) ~[hazelcast-tomcat7-sessionmanager-1.1.2.jar:na] at com.hazelcast.session.HazelcastSession.readData(HazelcastSession.java:127) ~[hazelcast-tomcat7-sessionmanager-1.1.2.jar:na] at com.hazelcast.internal.serialization.impl.DataSerializableSerializer.readInternal(DataSerializableSerializer.java:158) ~[hazelcast-all-3.9.jar:3.9] at com.hazelcast.internal.serialization.impl.DataSerializableSerializer.read(DataSerializableSerializer.java:105) ~[hazelcast-all-3.9.jar:3.9] at com.hazelcast.internal.serialization.impl.DataSerializableSerializer.read(DataSerializableSerializer.java:50) ~[hazelcast-all-3.9.jar:3.9] at com.hazelcast.internal.serialization.impl.StreamSerializerAdapter.read(StreamSerializerAdapter.java:48) ~[hazelcast-all-3.9.jar:3.9] at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toObject(AbstractSerializationService.java:185) ~[hazelcast-all-3.9.jar:3.9] at com.hazelcast.map.impl.proxy.MapProxySupport.toObject(MapProxySupport.java:1149) ~[hazelcast-all-3.9.jar:3.9] at com.hazelcast.map.impl.proxy.MapProxyImpl.remove(MapProxyImpl.java:212) ~[hazelcast-all-3.9.jar:3.9] at com.hazelcast.session.HazelcastSessionManager.remove(HazelcastSessionManager.java:328) ~[hazelcast-tomcat7-sessionmanager-1.1.2.jar:na] at com.hazelcast.session.HazelcastSessionManager.remove(HazelcastSessionManager.java:294) ~[hazelcast-tomcat7-sessionmanager-1.1.2.jar:na] at org.apache.catalina.session.StandardSession.expire(StandardSession.java:833) ~[catalina.jar:7.0.82] at org.apache.catalina.session.StandardSession.expire(StandardSession.java:732) ~[catalina.jar:7.0.82] at org.apache.catalina.session.StandardSession.invalidate(StandardSession.java:1264) ~[catalina.jar:7.0.82] at org.apache.catalina.session.StandardSessionFacade.invalidate(StandardSessionFacade.java:183) ~[catalina.jar:7.0.82] at org.springframework.security.web.authentication.logout.SecurityContextLogoutHandler.logout(SecurityContextLogoutHandler.java:65) ~[spring-security-web-4.0.2.RELEASE.jar:4.0.2.RELEASE]

[Jaromir Hamala]

Hello Hodaya,

it smells very much like this issue: https://github.com/hazelcast/hazelcast/issues/6781 and https://github.com/EsotericSoftware/kryo/issues/372

A long story short: During deserialization Kryo consumes too many bytes from the input stream and this corrupts the stream. This is solvable with chunking. See this as an example: https://github.com/jerrinot/subzero/blob/5f80142d77e4ba1759f71c9fe0b5fde7623ada8b/subzero-core/src/main/java/info/jerrinot/subzero/internal/strategy/KryoStrategy.java#L22

Cheers.

Jaromir

[hoda...@gmail.com]

Hi Jaromir,

Thanks for your answer, but I saw that the 2 issues in github are on java.io.EOFException, whereas my exception is "HazelcastSerializationException: There is no suitable de-serializer for type "

[Jaromir Hamala]

it could still be the same cause. The error is "There is no suitable de-serializer for type 16843028" and I assume you did not registered a serializer with ID 16843028. The number looks random. This indicates the ID is some random mambo-jambo - most likely a result of a corrupted stream.  And we know Kryo is prone to a stream corruption.

Cheers,

Jaromir

[Alparslan Avcı]

Hello Hodaya,

Hazelcast's Tomcat Based Web Session Replication only supports Serializable objects as Session objects when you need to cluster them. This requirement also applies when a global serializer is set up in the cluster configuration. Thus, you need to make your session objects Serializable to make it work.

You can also find this requirement in the documentation: https://github.com/hazelcast/hazelcast-tomcat-sessionmanager#features-and-requirements

I've raised a Github issue on the repo regarding this request, please track the process from there: https://github.com/hazelcast/hazelcast-tomcat-sessionmanager/issues/38

Regards,

Alparslan

[hoda...@gmail.com]

Hi Alparslan,

From Hazelcast documentation, I understood that if a class does not implement 'Serializable', it is serialized with the global serializer.

See here: http://docs.hazelcast.org/docs/latest-development/manual/html/Serialization/Serialization_Interface_Types.html

So I don't understand your comment above.

[hoda...@gmail.com]

Hi Jaromir,

I read the posts you suggested. I saw the exceptions started after Hazelcast was upgraded from 3.5 to 3.6 (although the problem seems to be in Kryo, but it looks like Hazelcast 3.5 was able to handle it).

So I tried to downgrade Hazelcast to 3.5.5 and it seemed to work. I removed Java Serialization from a few classes and there were no serialization exceptions (I checked only on one instance ).  

But, for one class it did not work and I received "com.hazelcast.nio.serialization.HazelcastSerializationException: java.io.NotSerializableException".

Why did I get this exception? A class without java seriailization should be handled by the global serializer.

[Alparslan Avcı]

Hi Hodaya,

Your statement is correct for regular Hazelcast cluster, but you are using another module: Tomcat Based Web Session Replication. When you use this module, you need to use Serializable objects as Session objects since the module won't work with the session objects that does not implement Serializable interface. This also applies when you set a global serializer on the cluster.

Regards,

Alparslan

[hoda...@gmail.com]

I could accept/understand this unless this weird scenario:

I removed java serialization from 2 classes. On one class (I'm almost sure was used) there was no exception, but on the other one, I got "HazelcastSerializationException: java.io.NotSerializableException"

[hoda...@gmail.com]

Hi Alparslan 

Thank you for your help! I think I understood what you said, so you can ignore my last question.

I'm trying now to find a way to catch Hazelcast exceptions when class is not Serializable, does Hazelcast have some kind of an exception handler for that?

[Alparslan Avcı]

Hi Hodaya,

Unfortunately, there is no specific exception handler for catching these exceptions, but you can try to catch  HazelcastSerializationException since this is the exception when Hazelcast faces an issue related to serialization.

Regards,

Alparslan