kryo issues - new release?

[Martin Grotzke]

Hi,

our project is targeting a new major release the next weeks and the
kryo version I integrated is still a patched one

I'm interested in these two issues/patches integrated:
#21: Better error reporting for FieldSerializer
#22: Provide possibility to set the RegisteredClass for a given type -
putRegisteredClass(Class, RegisteredClass)

Is it possible to pull them in and release a new kryo version?

Thanx && cheers,
Martin

[Nate]

Hi Martin,

Sorry things have been a bit slow lately. I'll get these in and make a release in the next couple days.

-Nate

--
You received this message because you are subscribed to the "kryo-users" group.
http://groups.google.com/group/kryo-users

[Martin Grotzke]

Hi Nate,

really great, thanx in advance!

Cheers,
Martin

--
Martin Grotzke
http://www.javakaffee.de/blog/

[Nate]

Can you take a look at the issues and let me know if the fixes work for you? If so, I'll make a new release.

-Nate

[Martin Grotzke]

I commented #21, #24 is not that important to me (and typing on my phone is a little bit hard).

Cheers,
Martin

[Martin Grotzke]

Hi Nate,

I attached a modified path to #21. If you're fine with this, the
original patch or some variant of this I'd be very happy about a new
release :-)

Thanx in advance,
cheers,
Martin

On Sat, Aug 14, 2010 at 1:42 PM, Martin Grotzke

[Nate]

1.02 is up. Let me know if the serialization trace stuff doesn't work for you.

-Nate

[Martin Grotzke]

Hi Nate,

great, I think your solution is exactly what I wanted! :-)

Unfortunately, when upgrading kryo in kryo-serializers and running
tests I get IllegalAccessExceptions, that I don't get with the
previous version (1.01).

At the end of the mail you find some stack traces from the test
http://github.com/magro/kryo-serializers/blob/kryo-1.02/src/test/java/de/javakaffee/kryoserializers/KryoTest.java

Thanx && cheers,
Martin

=====================================================================
FAILED: testClassSerializer
com.esotericsoftware.kryo.SerializationException: Unable to serialize
object of type: de.javakaffee.kryoserializers.TestClasses$Holder
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:515)
at com.esotericsoftware.kryo.ObjectBuffer.writeObject(ObjectBuffer.java:247)
at de.javakaffee.kryoserializers.KryoTest.serialize(KryoTest.java:621)
at de.javakaffee.kryoserializers.KryoTest.testClassSerializer(KryoTest.java:350)
Caused by: com.esotericsoftware.kryo.SerializationException: Error
accessing field: item
(de.javakaffee.kryoserializers.TestClasses$Holder)
at com.esotericsoftware.kryo.serialize.FieldSerializer.writeObjectData(FieldSerializer.java:175)
at com.esotericsoftware.kryo.serialize.ReferenceFieldSerializer.writeObjectData(ReferenceFieldSerializer.java:52)
at com.esotericsoftware.kryo.Serializer.writeObject(Serializer.java:43)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:511)
... 25 more
Caused by: java.lang.IllegalAccessException: Class
com.esotericsoftware.kryo.serialize.FieldSerializer$CachedField can
not access a member of class
de.javakaffee.kryoserializers.TestClasses$Holder with modifiers ""
at com.esotericsoftware.kryo.serialize.FieldSerializer$CachedField.get(FieldSerializer.java:302)
at com.esotericsoftware.kryo.serialize.FieldSerializer.writeObjectData(FieldSerializer.java:154)
... 28 more
... Removed 26 stack frames

FAILED: testSharedObjectIdentity("AtomicInteger", 42)
com.esotericsoftware.kryo.SerializationException: Unable to serialize
object of type: de.javakaffee.kryoserializers.TestClasses$HolderList
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:515)
at com.esotericsoftware.kryo.ObjectBuffer.writeObject(ObjectBuffer.java:247)
at de.javakaffee.kryoserializers.KryoTest.serialize(KryoTest.java:621)
at de.javakaffee.kryoserializers.KryoTest.testSharedObjectIdentity(KryoTest.java:402)
Caused by: com.esotericsoftware.kryo.SerializationException: Error
accessing field: holders
(de.javakaffee.kryoserializers.TestClasses$HolderList)
at com.esotericsoftware.kryo.serialize.FieldSerializer.writeObjectData(FieldSerializer.java:175)
at com.esotericsoftware.kryo.serialize.ReferenceFieldSerializer.writeObjectData(ReferenceFieldSerializer.java:52)
at com.esotericsoftware.kryo.Serializer.writeObject(Serializer.java:43)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:511)
... 25 more
Caused by: java.lang.IllegalAccessException: Class
com.esotericsoftware.kryo.serialize.FieldSerializer$CachedField can
not access a member of class
de.javakaffee.kryoserializers.TestClasses$HolderList with modifiers ""
at com.esotericsoftware.kryo.serialize.FieldSerializer$CachedField.get(FieldSerializer.java:302)
at com.esotericsoftware.kryo.serialize.FieldSerializer.writeObjectData(FieldSerializer.java:154)
... 28 more
... Removed 26 stack frames

FAILED: testInnerClass
com.esotericsoftware.kryo.SerializationException: Unable to serialize
object of type: de.javakaffee.kryoserializers.TestClasses$Container
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:515)
at com.esotericsoftware.kryo.ObjectBuffer.writeObject(ObjectBuffer.java:247)
at de.javakaffee.kryoserializers.KryoTest.serialize(KryoTest.java:621)
at de.javakaffee.kryoserializers.KryoTest.testInnerClass(KryoTest.java:358)
Caused by: com.esotericsoftware.kryo.SerializationException: Error
accessing field: this$1
(de.javakaffee.kryoserializers.TestClasses$Container$Body)
Serialization trace:
_body (de.javakaffee.kryoserializers.TestClasses$Container)
at com.esotericsoftware.kryo.serialize.FieldSerializer.writeObjectData(FieldSerializer.java:175)
at com.esotericsoftware.kryo.serialize.ReferenceFieldSerializer.writeObjectData(ReferenceFieldSerializer.java:52)
at com.esotericsoftware.kryo.serialize.FieldSerializer.writeObjectData(FieldSerializer.java:165)
at com.esotericsoftware.kryo.serialize.ReferenceFieldSerializer.writeObjectData(ReferenceFieldSerializer.java:52)
at com.esotericsoftware.kryo.Serializer.writeObject(Serializer.java:43)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:511)
... 25 more
Caused by: java.lang.IllegalAccessException: Class
com.esotericsoftware.kryo.serialize.FieldSerializer$CachedField can
not access a member of class
de.javakaffee.kryoserializers.TestClasses$Container$Body with
modifiers "final"
at com.esotericsoftware.kryo.serialize.FieldSerializer$CachedField.get(FieldSerializer.java:302)
at com.esotericsoftware.kryo.serialize.FieldSerializer.writeObjectData(FieldSerializer.java:154)
... 30 more
... Removed 26 stack frames

=====================================================================

[Nate]

Hi Martin,

Hmm, can you try again with my latest checkin? Had the setAccessible logic wrong. It had always been wrong, so when I shuffled it around it became apparent.

-Nate

[Martin Grotzke]

Hi Nate,

tests in kryo-serializers are fine now. When running tests with
kryo-serializers and the new kryo version in
memcached-session-manager, I have still one failing test:

FAILED: testDeserializeHibernateCollection
com.esotericsoftware.kryo.SerializationException: Unable to serialize
object of type: java.util.concurrent.ConcurrentHashMap

at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:515)
at com.esotericsoftware.kryo.ObjectBuffer.writeObject(ObjectBuffer.java:247)

at de.javakaffee.web.msm.serializer.kryo.KryoTranscoder.serializeAttributes(KryoTranscoder.java:268)
at de.javakaffee.web.msm.TranscoderService.serializeAttributes(TranscoderService.java:140)
at de.javakaffee.web.msm.TranscoderService.serialize(TranscoderService.java:85)
at de.javakaffee.web.msm.serializer.hibernate.AbstractHibernateCollectionsTest.testDeserializeHibernateCollection(AbstractHibernateCollectionsTest.java:93)
Caused by: com.esotericsoftware.kryo.SerializationException: Unable to
serialize object of type:
de.javakaffee.web.msm.serializer.hibernate.AbstractHibernateCollectionsTest$Person
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:484)
at com.esotericsoftware.kryo.serialize.MapSerializer.writeObjectData(MapSerializer.java:104)

at com.esotericsoftware.kryo.Serializer.writeObject(Serializer.java:43)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:511)

... 27 more
Caused by: com.esotericsoftware.kryo.SerializationException: Error
accessing field: animals
(de.javakaffee.web.msm.serializer.hibernate.AbstractHibernateCollectionsTest$Person)

at com.esotericsoftware.kryo.serialize.FieldSerializer.writeObjectData(FieldSerializer.java:175)
at com.esotericsoftware.kryo.serialize.ReferenceFieldSerializer.writeObjectData(ReferenceFieldSerializer.java:52)

at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:480)
... 30 more

Caused by: java.lang.IllegalAccessException: Class
com.esotericsoftware.kryo.serialize.FieldSerializer$CachedField can
not access a member of class

de.javakaffee.web.msm.serializer.hibernate.AbstractHibernateCollectionsTest$Person
with modifiers "public"

at com.esotericsoftware.kryo.serialize.FieldSerializer$CachedField.get(FieldSerializer.java:302)
at com.esotericsoftware.kryo.serialize.FieldSerializer.writeObjectData(FieldSerializer.java:154)

... 32 more

If it's of interest, here's the test (and superclass):
http://github.com/magro/memcached-session-manager/blob/master/kryo-serializer/src/test/java/de/javakaffee/web/msm/serializer/kryo/KryoTranscoderHibernateCollectionsTest.java
http://github.com/magro/memcached-session-manager/blob/master/core/src/test/java/de/javakaffee/web/msm/serializer/hibernate/AbstractHibernateCollectionsTest.java
(the latter one contains the Person class as you can see from the stacktrace)

Cheers,
Martin

[Nate]

Ah. Can you update and try again? :)

-Nate

[Martin Grotzke]

This still doesn't work:

FAILED: testDeserializeHibernateCollection
com.esotericsoftware.kryo.SerializationException: Unable to serialize
object of type: java.util.concurrent.ConcurrentHashMap
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:515)
at com.esotericsoftware.kryo.ObjectBuffer.writeObject(ObjectBuffer.java:247)
at de.javakaffee.web.msm.serializer.kryo.KryoTranscoder.serializeAttributes(KryoTranscoder.java:268)
at de.javakaffee.web.msm.TranscoderService.serializeAttributes(TranscoderService.java:140)
at de.javakaffee.web.msm.TranscoderService.serialize(TranscoderService.java:85)
at de.javakaffee.web.msm.serializer.hibernate.AbstractHibernateCollectionsTest.testDeserializeHibernateCollection(AbstractHibernateCollectionsTest.java:93)
Caused by: com.esotericsoftware.kryo.SerializationException: Unable to
serialize object of type:
de.javakaffee.web.msm.serializer.hibernate.AbstractHibernateCollectionsTest$Person
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:484)
at com.esotericsoftware.kryo.serialize.MapSerializer.writeObjectData(MapSerializer.java:104)
at com.esotericsoftware.kryo.Serializer.writeObject(Serializer.java:43)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:511)
... 27 more
Caused by: com.esotericsoftware.kryo.SerializationException: Error
accessing field: animals
(de.javakaffee.web.msm.serializer.hibernate.AbstractHibernateCollectionsTest$Person)

at com.esotericsoftware.kryo.serialize.FieldSerializer.writeObjectData(FieldSerializer.java:180)

at com.esotericsoftware.kryo.serialize.ReferenceFieldSerializer.writeObjectData(ReferenceFieldSerializer.java:52)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:480)
... 30 more
Caused by: java.lang.IllegalAccessException: Class
com.esotericsoftware.kryo.serialize.FieldSerializer$CachedField can
not access a member of class
de.javakaffee.web.msm.serializer.hibernate.AbstractHibernateCollectionsTest$Person
with modifiers "public"

at com.esotericsoftware.kryo.serialize.FieldSerializer$CachedField.get(FieldSerializer.java:307)
at com.esotericsoftware.kryo.serialize.FieldSerializer.writeObjectData(FieldSerializer.java:159)
... 32 more

... Removed 26 stack frames

I changed the FieldSerializer a little bit:
Index: src/com/esotericsoftware/kryo/serialize/FieldSerializer.java
===================================================================
--- src/com/esotericsoftware/kryo/serialize/FieldSerializer.java
(revision 121)
+++ src/com/esotericsoftware/kryo/serialize/FieldSerializer.java
(working copy)
@@ -75,11 +75,7 @@
if (Modifier.isStatic(modifiers)) continue;
if (field.isSynthetic() &&
ignoreSyntheticFields) continue;

- Class fieldClass = field.getType();
- boolean isFinal = isFinal(fieldClass);
- boolean isPublic =
Modifier.isPublic(modifiers) &&
Modifier.isPublic(fieldClass.getModifiers());
-
- if (!isPublic || isFinal) {
+ if (!field.isAccessible()) {
if (!setFieldsAsAccessible) continue;
try {
field.setAccessible(true);
@@ -96,10 +92,11 @@
cachedField.canBeNull = false;

// Always use the same serializer for this
field if the field's class is final.
- if (isFinal) cachedField.fieldClass = fieldClass;
+ final Class<?> fieldType = field.getType();
+ if (isFinal(fieldType)) cachedField.fieldClass = fieldType;

cachedFields.add(cachedField);
- if (isPublic) publicFields.add(cachedField);
+ if (Modifier.isPublic(modifiers) &&
Modifier.isPublic(fieldType.getModifiers()))
publicFields.add(cachedField);
}

if (!Util.isAndroid &&
Modifier.isPublic(type.getModifiers()) && !publicFields.isEmpty()) {

With this modification tests are fine: kryo tests, kryo-serializer
tests and msm-kryo-serializer tests...

What do you think?

Cheers,
Martin

[Nate]

Ha! Good find! XD Checked in. I'll make a new release in a bit.

-Nate

[Martin Grotzke]

Hi Nate,

I integrated all upgraded libs to our project but now I get strange
ClassNotFoundExceptions for reflectasm.FieldAccess:

SEVERE: An exception or error occurred in the container during the
request processing
java.lang.NoClassDefFoundError: com/esotericsoftware/reflectasm/FieldAccess
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
at java.lang.ClassLoader.defineClass(ClassLoader.java:466)
at com.esotericsoftware.reflectasm.AccessClassLoader.defineClass(AccessClassLoader.java:14)
at com.esotericsoftware.reflectasm.FieldAccess.get(FieldAccess.java:195)
at com.esotericsoftware.kryo.serialize.FieldSerializer.rebuildCachedFields(FieldSerializer.java:105)
at com.esotericsoftware.kryo.serialize.FieldSerializer.<init>(FieldSerializer.java:46)

I had also upgraded reflectasm to 0.9 (otherwise it wouldn't get so
far), and the lib is available in the classpath as all other jars.
Also I can't image that your changes in AccessClassLoader might cause this.

Do you have an idea about this?

Cheers,
Martin

[Martin Grotzke]

Hi Nate,

I debugged this a little bit by printing the name of the class that
should be defined by AccessClassLoader. The class that could not be
defined looks s.th. like this (it's a wicket page):

@SystemPage( humlid = "homepage", displayvalue = "Home" )
@MountQueryString
public class HomePage extends BasePage implements SomeMarkerInterface {

@SuppressWarnings( "hiding" )
public static final String MOUNT_PATH = Strings.EMPTY;

@SpringBean
private ConfigurationProvider _configurationProvider;

@SpringBean
private RequestContext _requestContext;

....

}

The BasePage contained the public field:

public class BasePage extends ContentPage {

@SuppressWarnings( "hiding" )
public static final SingleParameterNoPathCodingStrategy
URL_CODING_STRATEGY =
new SingleParameterNoPathCodingStrategy( Strings.EMPTY,
CarpetPage.class );

@SpringBean
public ContentPageService _contentPageService;

@SpringBean
private RequestContext _requestContext;

....

}

Not sure if this information helps...

I was able to remove this public field (it was really not needed) so
that reflectasm.FieldAccess is no longer used for this class - and
therefore the error is gone.
Still the question is what was causing the error.

Cheers,
Martin

On Fri, Aug 20, 2010 at 10:13 PM, Martin Grotzke

[Martin Grotzke]

Btw, fields annotated with @SpringBean are most probably jdk proxies (cglib proxies alternatively).

Cheers,
Martin

[Nate]

Can you get the latest reflectasm from SVN and try again? I believe I fixed a classloader problem there. It is difficult to test though, and you seem to have the test already. ;)

-Nate

[Martin Grotzke]

Of course :-)

Unfortunately I still have a NCDFE, but now for DerValueFieldAccess:

Defining class sun.security.util.DerValueFieldAccess
Aug 21, 2010 1:36:12 AM org.apache.catalina.connector.CoyoteAdapter service

SEVERE: An exception or error occurred in the container during the
request processing
java.lang.NoClassDefFoundError: com/esotericsoftware/reflectasm/FieldAccess
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
at java.lang.ClassLoader.defineClass(ClassLoader.java:466)

at com.esotericsoftware.reflectasm.AccessClassLoader.defineClass(AccessClassLoader.java:13)
at com.esotericsoftware.reflectasm.FieldAccess.get(FieldAccess.java:193)
...
Caused by: java.lang.ClassNotFoundException:
com.esotericsoftware.reflectasm.FieldAccess
at java.lang.ClassLoader.findClass(ClassLoader.java:359)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)

Cheers,
Martin

[Nate]

Ok, I added a test to reproduce the problem and fixed it in ReflectASM for sure this time. :)

-Nate

[Martin Grotzke]

Hi Nate,

really great, DerValueFieldAccess and lots of other stuff is defined
successfully now.

I'm still having issues with spring JdkDynamicAopProxy, but that's not
your business, I need to find a way to serialize a light
representation of them (tracking down this already independently from
kryo/reflectasm update).

Thanx for your fast help,
cheers,
Martin

[Siddharth Sharma]

Hi Nate and Martin,

I am still receiving NCDFE 

I am using

kryo-serializers-0.11.jar

kryo-1.04.jar

msm-kryo-serializer-1.8.1.jar

reflectasm-1.01.jar

Caused by: java.lang.NoClassDefFoundError: com/esotericsoftware/reflectasm/FieldAccess

at com.esotericsoftware.kryo.serialize.FieldSerializer.rebuildCachedFields(FieldSerializer.java:113)

at com.esotericsoftware.kryo.serialize.FieldSerializer.<init>(FieldSerializer.java:48)

at com.esotericsoftware.kryo.serialize.ReferenceFieldSerializer.<init>(ReferenceFieldSerializer.java:26)

at de.javakaffee.web.msm.serializer.kryo.ReferenceFieldSerializerFactory.newDefaultSerializer(ReferenceFieldSerializerFactory.java:34)

at com.saba.cache.KryoTranscoder$1.newDefaultSerializer(KryoTranscoder.java:219)

at com.esotericsoftware.kryo.Kryo.newSerializer(Kryo.java:273)

at com.saba.cache.KryoTranscoder$1.newSerializer(KryoTranscoder.java:192)

at com.esotericsoftware.kryo.Kryo.handleUnregisteredClass(Kryo.java:329)

at com.saba.cache.KryoTranscoder$1.handleUnregisteredClass(KryoTranscoder.java:213)

at com.esotericsoftware.kryo.Kryo.getRegisteredClass(Kryo.java:305)

at com.esotericsoftware.kryo.Kryo.writeClass(Kryo.java:374)

at com.esotericsoftware.kryo.serialize.FieldSerializer.writeObjectData(FieldSerializer.java:173)

at com.esotericsoftware.kryo.serialize.ReferenceFieldSerializer.writeObjectData(ReferenceFieldSerializer.java:52)

at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:489)

at com.esotericsoftware.kryo.serialize.CollectionSerializer.writeObjectData(CollectionSerializer.java:86)

at com.esotericsoftware.kryo.serialize.FieldSerializer.writeObjectData(FieldSerializer.java:175)

at com.esotericsoftware.kryo.serialize.ReferenceFieldSerializer.writeObjectData(ReferenceFieldSerializer.java:52)

at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:489)

at com.esotericsoftware.kryo.ObjectBuffer.writeClassAndObject(ObjectBuffer.java:230)

at com.saba.cache.KryoTranscoder.serialize(KryoTranscoder.java:340)

at net.spy.memcached.transcoders.SerializingTranscoder.encode(SerializingTranscoder.java:162)

at net.spy.memcached.MemcachedClient.asyncStore(MemcachedClient.java:296)

at net.spy.memcached.MemcachedClient.set(MemcachedClient.java:884)

Caused by: java.lang.ClassNotFoundException: com.esotericsoftware.reflectasm.FieldAccess

at java.net.URLClassLoader$1.run(URLClassLoader.java:366)

at java.net.URLClassLoader$1.run(URLClassLoader.java:355)

at java.security.AccessController.doPrivileged(Native Method)

at java.net.URLClassLoader.findClass(URLClassLoader.java:354)

at java.lang.ClassLoader.loadClass(ClassLoader.java:423)

at java.lang.ClassLoader.loadClass(ClassLoader.java:356)

What am I doing wrong?

Thanks,

SId