Encountered unregistered class ID - CompatibleSerializer

[Venu Kemthur]

We are using CompatibleSerializer and we see that in few cases Kryo fails to deserialize the data that it serialized.

(Serialized data is written into memcache)

com.esotericsoftware.kryo.KryoException: Encountered unregistered class ID: 65

        at com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:119)

        at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:656)

        at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:767)

How should one go about debugging this?

I enabled Kryo tracing and see this 

00:01 TRACE: [kryo] Write initial object reference 65: jpg

00:01 TRACE: [kryo] Write: jpg

Am I right in mapping class ID 65 in exception trace to this output?

Any pointers on how to analyze ' unregistered class ID' would be helpful.

Thanks

Venu

[mongonix]

Hi Venu,

On Wednesday, December 4, 2013 6:16:03 AM UTC+1, Venu Kemthur wrote:

We are using CompatibleSerializer and we see that in few cases Kryo fails to deserialize the data that it serialized.

(Serialized data is written into memcache)

com.esotericsoftware.kryo.KryoException: Encountered unregistered class ID: 65

        at com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:119)

        at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:656)

        at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:767)

How should one go about debugging this?

I enabled Kryo tracing and see this 

00:01 TRACE: [kryo] Write initial object reference 65: jpg

00:01 TRACE: [kryo] Write: jpg

Am I right in mapping class ID 65 in exception trace to this output?

No, 65 is the number of your object in the internal reference table. Kryo has to remember all objects it has seen so far.

Overall, the idea of looking into trace output is the right one, but based only on the provided trace-snippet it is a bit difficult to help you.

Normally, Kryo logs something like "Write class" if a remember correctly. It also logs a lot of info about which field of a class is being serialized.

If you want us to look into this issue, could you provide either a more complete trace or even better a self-contained reproducible test-case?

Any pointers on how to analyze ' unregistered class ID' would be helpful.

See above. The immediate reason for this message is that Kryo reads bytes at the wrong place, i.e. it expects a class id at this place, but reads the rest of a previous object or something like this.

Usually such a problem happens in the following situations situations:

- your Kryo instances for serialization and deserialization are configured differently

- you use different methods to read/write objects, e.g. writeObject and then readClassAndObject instead of readObject

- your custom serializer misbehaves

- there is a bug in Kryo

 

-Leo

[Venu Kemthur]

Thanks Leo.

Here is how we have configured kryo (using CompatibleFieldSerializer) -

 

private static ThreadLocal<Kryo> compatibleKryo = new ThreadLocal<Kryo>() {

        @Override

        protected Kryo initialValue() {

            Kryo kryo = new Kryo();

            kryo.setDefaultSerializer(CompatibleFieldSerializer.class);

            return kryo;

        }

    };

and we have a custom transcoder for memcache which serializes and deserializes cache data using kryo.

We don't register any classes explicitly and rely on Kryo implicit registration. And we are seeing that in few cases, we see Kryo write happening fine (checked via Kryo trace), but when that same data is being read by Kryo it fails with this error (we use readClassAndObject and writeClassAndObject)

We also see a pattern in these errors where kryo fails to read what it wrote. The data typically has either html entities (&amp;) or Unicode (non english characters)

in fact in one of the cases, we started skipping I18n data and kryo deserialization started working!

Is there something specific to be done in terms of handling these? (Does Kryo interpret & etc in some other way?)

Looked at the StringSerailizer in Kryo and it seems to be handing Unicode fine.

Thanks

Venu

[mongonix]

On Wednesday, December 4, 2013 11:57:17 PM UTC+1, Venu Kemthur wrote:

Thanks Leo.

Here is how we have configured kryo (using CompatibleFieldSerializer) -

 

private static ThreadLocal<Kryo> compatibleKryo = new ThreadLocal<Kryo>() {

        @Override

        protected Kryo initialValue() {

            Kryo kryo = new Kryo();

            kryo.setDefaultSerializer(CompatibleFieldSerializer.class);

            return kryo;

        }

    };

This seems OK for me.

 

and we have a custom transcoder for memcache which serializes and deserializes cache data using kryo.

We don't register any classes explicitly and rely on Kryo implicit registration. And we are seeing that in few cases, we see Kryo write happening fine (checked via Kryo trace), but when that same data is being read by Kryo it fails with this error (we use readClassAndObject and writeClassAndObject)

We also see a pattern in these errors where kryo fails to read what it wrote. The data typically has either html entities (&amp;) or Unicode (non english characters)

in fact in one of the cases, we started skipping I18n data and kryo deserialization started working!

Is there something specific to be done in terms of handling these? (Does Kryo interpret & etc in some other way?)

Looked at the StringSerailizer in Kryo and it seems to be handing Unicode fine.

In principle, Kryo should process unicode just fine. There are no known bugs related to that.

But it is difficult to answer your question without a reproducible test-case.  Most likely, you can reproduce it without any external dependencies like memcache, by simply using two different Kryo instances and writing/reading the same data that you do in your real use-case.

Of course, I don't know if you are allowed to provide us with a test-case or if your company policy forbids it, because it could disclose something ...

-Leo

[Venu Kemthur]

Thanks.

I'll try to simulate our production code and try to get a reproducible test case.

I tried doing kryo.setReferences(false) (defaults to true)

and now instead of class ID and seeing the class name

2013-12-08 14:26:30.423 ERROR (KryoSerializingTranscoder.java:70) [http-apr-0.0.0.0-4080-exec-10]: Error deserializing using CompatibleSerializer: Unable to find class: ^A^A^C^XThe_Shawshank_Redemption

com.esotericsoftware.kryo.KryoException: Unable to find class: ^A^A^C^XThe_Shawshank_Redemption

        at com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:138)

        at com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:115)

        at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:656)

        at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:767)

caused by: java.lang.ClassNotFoundException: ^A^A^C^XThe_Shawshank_Redemption

        at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1714)

        at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1559)

        at java.lang.Class.forName0(Native Method)

        at java.lang.Class.forName(Class.java:270)

        at com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:136)

Not sure of the control characters, but the class that it is trying to load is actually a 'String' within a Bean that we try to deserialize. What would lead kryo to read something like this as a 'class' and not 'object'?

Thanks

Venu 

[Venu Kemthur]

Sorry for the typos in my earlier mail. I meant to say that 'The_Shawshank_redemption' is a string value that kryo serializes, but happens to read it as class (and not object) when deserializing.

So we basically have a nested class like this

class Foo{

List<Bean> beanList;

}

class Bean

{

String name; // name here is The_Shawshank_redemption

}

we try to serialize and deserialize Foo

(I'll try to get a reproducible test code - meanwhile if you think something is going wrong with our usage of kryo, please let know)

-- Venu

[Chetan Narsude]

Can you send the exact declaration of Foo and Bean? A clue can be in the class declaration (inheritance)?

--
You received this message because you are subscribed to the "kryo-users" group.
http://groups.google.com/group/kryo-users
---
You received this message because you are subscribed to the Google Groups "kryo-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kryo-users+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Venu Kemthur]

this is standard composition. Bean is part of Foo class as i indicated earlier. 

let me model these as say movies

public class Movies implement Serializable{

 List<Movie> watchedMovies;

...

}

class Movie implements Serializable{

String movieName;

Date watchDate;

}

So Movies is being serialized; write happens fine - when the same is read back, a movie name set on Movie is being read as a class

Note movie name here had _ (not sure if it matters)

we write byte[] and read as byte[]

-- venu

[Joachim Durchholz]

This looks like Kryo is confused. Possible causes:

1) The serialized stream was written by a different version of Kryo than
you're using to read it.
2) A bug in Kryo (nested classes are somewhat special).
3) Something interfering with the data stream, e.g. something that
recodes UTF-8 to Latin-1.
4) There's also the possibility that something odd is going on with your
nested class. If I recall correctly, nested class name mangling isn't
standardized, so if the Kryo stream was written by a different kind of
JVM than what's running your Tomcat, you might be in trouble, too. (This
is a rather vague possibility, I just mention it in case your setup is
way more complicated than usual.)

If it's not (1), you'll need to strip the code to the minimum that can
reproduce the problem to find the cause.

[Venu Kemthur]

On Sunday, 8 December 2013 22:40:17 UTC-8, Toolforger wrote:

This looks like Kryo is confused. Possible causes:

1) The serialized stream was written by a different version of Kryo than
you're using to read it.

Same version is used to serialize/deserialize

 

2) A bug in Kryo (nested classes are somewhat special).

 

3) Something interfering with the data stream, e.g. something that
recodes UTF-8 to Latin-1.

this is possible - we are definitely seeing a pattern of this failing when data is i18n. but kryo serialized byte buffer is being handed over to Xmemcached client to write to memcached and we are suspecting if this is corrupting the byte stream (code walkthrough of this client doesn't seem to suggest anything )

  

4) There's also the possibility that something odd is going on with your
nested class. If I recall correctly, nested class name mangling isn't
standardized, so if the Kryo stream was written by a different kind of
JVM than what's running your Tomcat, you might be in trouble, too. (This
is a rather vague possibility, I just mention it in case your setup is
way more complicated than usual.)

No its the same JVM which serializes/deserializes - 

If it's not (1), you'll need to strip the code to the minimum that can
reproduce the problem to find the cause.

Yes I have tried that - i have mocked data on file - I read this using the same transcoder that our code uses and write to memcached.using the same client. 

This seems to work always (not sure if it matters - only that i am doing this test on Mac and issue we are seeing is on linux)

[Joachim Durchholz]

Am 09.12.2013 22:03, schrieb Venu Kemthur:
>
>> 3) Something interfering with the data stream, e.g. something that
>> recodes UTF-8 to Latin-1.
>>
> this is possible - we are definitely seeing a pattern of this failing when
> data is i18n. but kryo serialized byte buffer is being handed over to
> Xmemcached client to write to memcached and we are suspecting if this is
> corrupting the byte stream (code walkthrough of this client doesn't seem to
> suggest anything )
>

> This seems to work always (not sure if it matters - only that i am doing
> this test on Mac and issue we are seeing is on linux)

Are you using byte or character streams from the Java level?
Character streams recode to/from whatever is Java's idea of the platform
default, and recoding isn't always lossless.

It's best to inspect the bytes of the streams directly, i.e. in
hexadecimal - any kind of character display might have an encoding step
somewhere.
Not sure how to best do this on Mac, on Linux it would be the 'od' command.

Transferring data from one machine to the next might inadvertently get
you recoding, too, so it's best to look at the data on the machine where
it originated.