Question: Stack gets very large when serializing objects with circular references

[George Brova]

Hi,

When I try to serialize some objects which contain circular references, the stack gets very large (everything fits in heap<4000M, but the stack grows to >200M). Is this normal?

I am doing kryo.setReferences(true), and making the stack size big enough lets the program terminate normally.  With a smaller stack size, I get a StackOverflowError, like below. I am using kryo-2.21, but the same thing happens with kryo-2.22-SNAPSHOT.

Thanks for any thoughts,
George

Exception in thread "main" java.lang.StackOverflowError
    at sun.misc.Unsafe.getObject(Native Method)
    at sun.misc.Unsafe.getObject(Unsafe.java:249)
    at sun.reflect.UnsafeObjectFieldAccessorImpl.get(UnsafeObjectFieldAccessorImpl.java:37)
    at java.lang.reflect.Field.get(Field.java:372)
    at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:552)
    at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
    at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568)
    at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:75)
    at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:18)
    at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568)
    at com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:86)
    at com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:17)
    at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:501)
    at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:564)
    at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
    at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568)
    at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:75)
    at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:18)
    at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568)
    at com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:86)
    at com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:17)
    at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:501)
    at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:564)
...

[Nate]

Hi George,

I assume your object graph is very deep?

-Nate

--
You received this message because you are subscribed to the "kryo-users" group.
http://groups.google.com/group/kryo-users
---
You received this message because you are subscribed to the Google Groups "kryo-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kryo-users+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[mongonix]

I guess this is a perfect use-case, where my continuations-based serialization hack that I posted some time ago would be the savior :-)

Too bad that this approach required intrusive changes to Kryo, which often negatively affect performance (20-30%) in typical use-cases that do not use deep object graphs.

May be I should have a second try once the Unsafe-polishing is over. I guess with some refactoring it could be possible to use continuations only conditionally, when explicitly stated by the user.

-Leo

[George Brova]

Thank you both for the response. 

Nate, I don't think my object graph is deeper than about 20 objects. If I remove some references to get rid of cycles (but the depth of objects stays the same), the stack problems disappear. 

Leo, thanks for the suggestion. I tried applying your patch from March 12, but seem to be having the exact same problem. 

I'll try to investigate a little more tonight, and I'll write back. 

You received this message because you are subscribed to a topic in the Google Groups "kryo-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/kryo-users/op81RmHXOsM/unsubscribe?hl=en-US.
To unsubscribe from this group and all its topics, send an email to kryo-users+...@googlegroups.com.

[George Brova]

I think I figured this out, and Nate was right about the object graph being deep. 

I was trying to serialize a Graph object, with each Node object storing a Set<Node> of neighbors. Although I didn't realize at first, this actually results in a very deep object graph since objects are serialized depth-first. 

Is there an easy way to encourage Kryo to first create references to all my Node objects, without recursively serializing all their fields? 

Thanks,

George

[mongonix]

On Friday, April 26, 2013 10:21:55 PM UTC+2, George Brova wrote:

Thank you both for the response. 

Nate, I don't think my object graph is deeper than about 20 objects. If I remove some references to get rid of cycles (but the depth of objects stays the same), the stack problems disappear. 

Leo, thanks for the suggestion. I tried applying your patch from March 12, but seem to be having the exact same problem. 

This is interesting. I've serialized objects with depth of 800000. Therefore I'd like to get your example as a test for my continuation-based patch, if possible. Would it be possible to provide your code or at least the part of it that builds such deeply nested objects?

Thanks,

  Leo

[George Brova]

Hi Leo, I'd be happy to send you some sample code.  Where is the best place for me to upload that? (or should I just attach it to this email?)

[mongonix]

On Saturday, April 27, 2013 7:49:36 AM UTC+2, George Brova wrote:

Hi Leo, I'd be happy to send you some sample code.  Where is the best place for me to upload that? (or should I just attach it to this email?)

Hi George, I think the best place would be to attach it the corresponding issue, i.e. at http://code.google.com/p/kryo/issues/detail?id=103

Thanks,

  Leo

[Nate]

On Fri, Apr 26, 2013 at 11:27 PM, George Brova <gbr...@gmail.com> wrote:

I think I figured this out, and Nate was right about the object graph being deep. 

I was trying to serialize a Graph object, with each Node object storing a Set<Node> of neighbors. Although I didn't realize at first, this actually results in a very deep object graph since objects are serialized depth-first. 

Is there an easy way to encourage Kryo to first create references to all my Node objects, without recursively serializing all their fields? 

In your graph object you could serialize all the nodes using a serializer that doesn't write the neighbors set (see FieldSerializer#removeField if using that). Then serialize all the nodes again with a serializer that only writes the neighbors set. The last part would result in a node reference, then the set, which would only contain node references.

To improve this you could change the second part to write out only the sets to avoid a reference map look up to find the node that owns the set. This would assume you have some way of looking up a node from an index (the index being the order the sets are found). The reference map is a cuckoo map, so a get is at worst about 3 hash computations and 3 array look ups. The savings by avoiding some node references in the second part is mostly about reducing serialized size and the cost of writing and reading those bytes.

-Nate

[George Brova]

Leo, I tried again on a simpler example and actually your patch does fix the problem. Sorry for the confusion. 

Nate, thanks a lot for the explanation.  I'll run performance tests on Leo's patched code and probably use that if it's reasonably fast.  Though I really appreciate your explanation - I now have a much better understanding of what's going on. 

Thanks,

George

[Joachim Durchholz]

Am 26.04.2013 08:31, schrieb mongonix:
> I guess this is a perfect use-case, where my continuations-based
> serialization hack that I posted some time ago would be the savior :-)
> Too bad that this approach required intrusive changes to Kryo, which often
> negatively affect performance (20-30%) in typical use-cases that do not use
> deep object graphs.

Okay, I just finished applying the original patch (required some manual
labor, but the tests still succeed).

Non-continuation timings:
SerializationBenchmarkTest 147.421
testJavaSerialization 19.63
testJavaSerializationWithoutTryCatch 18.979
testKryoSerialization 21.752
testKryoSerializationUnmodified 23.291
testKryoSerializationWithoutTryCatch 22.141
testKryoSerializationWithoutTryCatchWithFastStreams 21.552
testKryoUnsafeSerializationWithoutTryCatch 6.659
testKryoUnsafeSerializationWithoutTryCatchWithoutAsm 6.96
testKryoUnsafeSerializationWithoutTryCatchWithoutReferences 6.457

Continuation timings:
SerializationBenchmarkTest 147.531
testJavaSerialization 19.558
testJavaSerializationWithoutTryCatch 18.914
testKryoSerialization 22.271
testKryoSerializationUnmodified 23.222
testKryoSerializationWithoutTryCatch 21.852
testKryoSerializationWithoutTryCatchWithFastStreams 21.77
testKryoUnsafeSerializationWithoutTryCatch 6.799
testKryoUnsafeSerializationWithoutTryCatchWithoutAsm 6.589
testKryoUnsafeSerializationWithoutTryCatchWithoutReferences 6.556

The differences are minimal, possibly just noise.
I suspected that the continuation stuff isn't even used, but no, the
FieldSerializationContinuation writers are instantiated and used for
writing, so that's probably not the cause.

So... does anybody have code where continuations are, indeed, slower?

[Nate]

Someone should update jvm-serializers to use the latest Kryo. Those micro benchmarks have had a lot more eyes on them.

-Nate

--
You received this message because you are subscribed to the "kryo-users" group.
http://groups.google.com/group/kryo-users
--- You received this message because you are subscribed to the Google Groups "kryo-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kryo-users+unsubscribe@googlegroups.com.

[mongonix]

On Friday, April 26, 2013 12:09:49 PM UTC+2, Toolforger wrote:

Am 26.04.2013 08:31, schrieb mongonix:
> I guess this is a perfect use-case, where my continuations-based
> serialization hack that I posted some time ago would be the savior :-)
> Too bad that this approach required intrusive changes to Kryo, which often
> negatively affect performance (20-30%) in typical use-cases that do not use
> deep object graphs.

Okay, I just finished applying the original patch (required some manual
labor, but the tests still succeed).

Thanks for trying out this patch!

Well, while developing the patch, I myself have seen cases where continuations are slower. I need to find them again, but from what I remember the use-cases did not use any deeply-nested object graphs and instead used classes with a lot of fields or arrays with a lot of elements or graphs with a lot of arrays. The slowdown came from the following:

- creation cost of continuation objects

- manipulations of the continuations stack and serializer stack

As you can see these are the actions that are not performed by the normal version of Kryo, i.e. this is an additional overhead. And of you execute those actions very often (i.e. you have a lot of objects in your object graph) than this overhead sums up. 

That being said, for any scenarios where the ultimate speed is not the main concern, the overhead of continuations-based approach will be minimal or even almost invisible.

BTW, I have a more up-to-date version of the continuations implementation. It fixes some bugs, allocates less continuations objects and is better optimized. If there is an interest, I may try to produce an improved patch in the future. Any wishes and comments regarding the approach and proposed continuations-based style of implementing serializers are welcome!

-Leo

[mongonix]

On Tuesday, April 30, 2013 12:17:11 PM UTC+2, Nate wrote:

Someone should update jvm-serializers to use the latest Kryo. Those micro benchmarks have had a lot more eyes on them.

This is a good idea. I second this proposal.

BTW, it may happen that Avro also will update their benchmarks :-) ... because I more or less ported my Unsafe-based serialization improvements to Avro. It will be merged into trunk soon. These improvements make reflection-based Avro serializers 3-5 times faster on average than they were before.

 

-Leo

To unsubscribe from this group and stop receiving emails from it, send an email to kryo-users+...@googlegroups.com.

[Nate]

On Tue, Apr 30, 2013 at 1:18 PM, mongonix <romi...@gmail.com> wrote:

BTW, it may happen that Avro also will update their benchmarks :-) ... because I more or less ported my Unsafe-based serialization improvements to Avro. It will be merged into trunk soon. These improvements make reflection-based Avro serializers 3-5 times faster on average than they were before.

Lame! ;)

-Nate

[Joachim Durchholz]

Am 30.04.2013 13:14, schrieb mongonix:
> BTW, I have a more up-to-date version of the continuations implementation.
> It fixes some bugs, allocates less continuations objects and is better
> optimized. If there is an interest, I may try to produce an improved patch
> in the future. Any wishes and comments regarding the approach and proposed
> continuations-based style of implementing serializers are welcome!

Yes, please publish as soon as you have something.

It would be interesting to see the code that's slower with continuations
than with the traditional approach.

[Brian Erst]

Has there been any movement on trying to get continuations into trunk? I'm bumping into the same sort of ArrayOutOfBounds issues in IdentiyObjectInMap and/or stack overflows on nested graphs. While they are big messages, the built-in Java serializer as well as a custom serializer written by my company have no problems serializing the objects. I'm using a combination of CompatibleFieldSerializer (for message wrappers) and TaggedFieldSerializer (for message payloads and backward/forward compatibility) and have cut my message size in half to prevent Kryo from erroring out. (The serialized bytes come to about 21Meg on the half-size messages).

IdentityObjectIntMap#resize silently throws an exception trying to resize the valueTable - it's caught in a finally block higher up within Kryo#writeObjectAndClass, but now IdentiyObjectIntMap has an inconsistent internal state (capacity is now twice as large as the actual valueTable array) and throws an uncaught exception when the IdentityObjectInMap#clean is called later on.

[mongonix]

On Thursday, August 15, 2013 4:55:06 PM UTC+2, Brian Erst wrote:

Has there been any movement on trying to get continuations into trunk? I'm bumping into the same sort of ArrayOutOfBounds issues in IdentiyObjectInMap and/or stack overflows on nested graphs.

I'll provide tomorrow an updated patch for continuations based on the current trunk and containing some bug fixes. But the overall code structure of the patch hasn't changed a lot since I posted the original version of continuations a while ago.

As for including this continuations-based implemnentation into trunk - this is a more difficult quesiton. The main problem is that this implementation is rather different from the current version of Kryo. The two main changes in Kryo code are:

1) Extending Kryo.java to support continuations - this change could be probably easily modularized by moving the changes into a dedicated class derived from Kryo and overloading some methods. If this new class would be used, it would automatically make use of continuations-based approach. 

2) Serializers - many of them are affected to support continuations. In a perfect world, we would like to be able to select between the traditional version of serializers in Kryo and continuations-based serializers. But the big question is how to do it in a modular way? I haven't spent too much time thinking about it yet. And I'm all open for any good proposals. 

If we find such a modular way, then integration of continuations into trunk should be pretty easy as it would be mostly orthogonal to the current code base and one could easily switch between a traditional and continuations-based versions by means of run-time Kryo configuration.

 

While they are big messages, the built-in Java serializer as well as a custom serializer written by my company have no problems serializing the objects. I'm using a combination of CompatibleFieldSerializer (for message wrappers) and TaggedFieldSerializer (for message payloads and backward/forward compatibility) and have cut my message size in half to prevent Kryo from erroring out. (The serialized bytes come to about 21Meg on the half-size messages).

It is actually quite interesting that only Kryo fails in this situations. Usually, for really deeply nested object graphs all approaches fail. 

Would it be possible for you to provide a test-case that demonstrates this behavior? May be this problem is an indication for some further issues in Kryo besides deep nesting levels?

 

IdentityObjectIntMap#resize silently throws an exception trying to resize the valueTable - it's caught in a finally block higher up within Kryo#writeObjectAndClass, but now IdentiyObjectIntMap has an inconsistent internal state (capacity is now twice as large as the actual valueTable array) and throws an uncaught exception when the IdentityObjectInMap#clean is called later on.

This way of handling resizing problems in Kryo seems to be pretty error-prone anyway. We should be more careful in such cases.

-Leo

[Nate]

On Thu, Aug 15, 2013 at 4:55 PM, Brian Erst <brian...@gmail.com> wrote:

IdentityObjectIntMap#resize silently throws an exception trying to resize the valueTable - it's caught in a finally block higher up within Kryo#writeObjectAndClass, but now IdentiyObjectIntMap has an inconsistent internal state (capacity is now twice as large as the actual valueTable array) and throws an uncaught exception when the IdentityObjectInMap#clean is called later on.

I don't see anywhere Kryo eats an exception. Can you be more specific?

It seems the cuckoo hashing is not working well for your objects. If your objects have poor hashcodes, IdentityObjectIntMap will have many collisions and resize larger than it would otherwise. You could try tweaking IdentityObjectIntMap, eg increase stashCapacity and/or pushIterations.

I suggest implementing your own MapReferenceResolver using java.util.IdentityHashMap (copypasta the existing one). The cuckoo hashing can have poor put performance, so maybe it isn't the best for for writtenObjects in MapReferenceResolver, which is likely 50/50 more gets/puts for most object graphs. The benefits of IdentityObjectIntMap are 1) no allocations for puts, 2) faster gets, 3) no boxing. The downside is slower puts and large backing arrays when there are hash collisions.

-Nate

 

[Brian Erst]

On Thu, Aug 15, 2013 at 2:29 PM, Nate <nathan...@gmail.com> wrote:

On Thu, Aug 15, 2013 at 4:55 PM, Brian Erst <brian...@gmail.com> wrote:

IdentityObjectIntMap#resize silently throws an exception trying to resize the valueTable - it's caught in a finally block higher up within Kryo#writeObjectAndClass, but now IdentiyObjectIntMap has an inconsistent internal state (capacity is now twice as large as the actual valueTable array) and throws an uncaught exception when the IdentityObjectInMap#clean is called later on.

I don't see anywhere Kryo eats an exception. Can you be more specific?

I should have been a little more precise in my answer. What happens is that the true exception is hidden. Code is 2.21.

IdentityObjectIntMap#resize:410 throws an OutOfMemoryException when it tries to resize the valueTable. At this point, the internal state of IdentityObjectIntMap is inconsistent - all the size-related fields (capacity, stashsize, etc.) are set to their new values before the valueTable has actually been resized.

This bubbles up to Kryo#writeClassAndObject:570, the finally clause of the try/finally block. It calls Kryo#reset which in turn calls IdentityObjectIntMap#clear. It attempts to iterate backward from the "end" of the keyTable to clear the keys by pointing to keyTable[capacity+stashsize], but the keyTable is not actually that large (half the size) due to the exception thrown in #resize. Now, an ArrayOutOfBounds exception is thrown and "overrides" the original OOMException - what you see in your stack trace is the ArrayOutOfBoundsException and you then need to start digging thru the Kryo code instead of figuring out if you've allocated enough memory space.

 

It seems the cuckoo hashing is not working well for your objects. If your objects have poor hashcodes, IdentityObjectIntMap will have many collisions and resize larger than it would otherwise. You could try tweaking IdentityObjectIntMap, eg increase stashCapacity and/or pushIterations.

I suggest implementing your own MapReferenceResolver using java.util.IdentityHashMap (copypasta the existing one). The cuckoo hashing can have poor put performance, so maybe it isn't the best for for writtenObjects in MapReferenceResolver, which is likely 50/50 more gets/puts for most object graphs. The benefits of IdentityObjectIntMap are 1) no allocations for puts, 2) faster gets, 3) no boxing. The downside is slower puts and large backing arrays when there are hash collisions.

-Nate

Yeah, I think there is something odd about the hashing of my objects. It would seem that how I do hashes shouldn't matter as you are using System#identityHashCode but I do seem to see a few resizes that are triggered by the stash running out of space, which if I understand the code correctly, would appear to indicate a bunch of hash collisions.

[Nate]

On Fri, Aug 16, 2013 at 6:03 PM, Brian Erst <brian...@gmail.com> wrote:

I should have been a little more precise in my answer. What happens is that the true exception is hidden.

Ah, yeah, once OOM is thrown all bets are off.

 

Yeah, I think there is something odd about the hashing of my objects. It would seem that how I do hashes shouldn't matter as you are using System#identityHashCode but I do seem to see a few resizes that are triggered by the stash running out of space, which if I understand the code correctly, would appear to indicate a bunch of hash collisions.

Oh you're right of course, it's an identity map so your hashcodes don't matter. I wonder if the changes to the internal hash2 and hash3 help (they were using a long and then truncating). I also wonder if we would have better hashing if we hash the identity hashcode instead of using it as-is for the first hash.

-Nate

[Brian Erst]

On Friday, August 16, 2013 11:38:11 AM UTC-5, Nate wrote:

Oh you're right of course, it's an identity map so your hashcodes don't matter. I wonder if the changes to the internal hash2 and hash3 help (they were using a long and then truncating). I also wonder if we would have better hashing if we hash the identity hashcode instead of using it as-is for the first hash.

-Nate

I implemented a very simplistic MapReferenceResolver (just using HashMap<Object, Integer>) and it seemed to work just fine. I spit out a count of objects being written and read and it was about 1.8M objects - the IdentityObjectIntMap was dying trying to allocate a 32M integer table, so something degenerate is going on.

[Nate]

Hope you meant IdentityHashMap. :)

The default load factor for IdentityObjectIntMap is 80%. The map resizes at this point to prevent slow puts. In my tests it tends to resize between 70% and 80%, which is an ok balance of memory/speed. It also means you should be able to store 10M entries with a POT backing size of ~16M (2^24). I'd like to know why your 1.8M entries are trying to allocate a 32M backing array! You can try adding this in IdentityObjectIntMap#resize:
System.out.println("resizing: " + size / (float)capacity);

--

[Brian Erst]

I quickly learned the errors of my ways of not using IdentityHashMap. :D  ("Why the hell is this enum serializing as 1 and deserializing as 3???")

I moved back to using my "full size" messages, which end up calling CustomReferenceResolver#addWriteObject about 2.8M times per message and it's just happily churning away.

If I have time to build a local copy of kryo and play with it, I'll look into why the IdentityObjectIntMap gets so carried away - at the point where it OOMd, it was trying for a 32M table on my "half size" messages and I don't think that was going to be the final resize - looking at some of the objects being serialized, it hadn't hit the end of the message.

FWIW - Here's my "custom" reference resolver.

    private static class CustomReferenceResolver implements ReferenceResolver
    {
        Kryo kryo;
        IdentityHashMap<Object, Integer> writeObjects = new IdentityHashMap<Object, Integer>();
        ArrayList                        readObjects  = new ArrayList();

        /**
         * Sets the Kryo instance that this ClassResolver will be used for. This is called automatically by Kryo.
         */
        @Override
        public void setKryo(Kryo kryo)
        {
            this.kryo = kryo;
        }

        /**
         * Returns an ID for the object if it has been written previously, otherwise returns -1.
         */
        @Override
        public int getWrittenId(Object object)
        {
            Integer i = writeObjects.get(object);
            if (i == null)
                return -1;
            else
                return i;
        }

        /**
         * Returns a new ID for an object that is being written for the first time.
         *
         * @return The ID, which is stored more efficiently if it is positive and must not be -1 or -2.
         */
        @Override
        public int addWrittenObject(Object object)
        {
            Integer id = new Integer(writeObjects.size());
            if (id % 1000 == 0)
                System.out.println("writeObject.size=" + id);
            writeObjects.put(object, id);
            return id;
        }

        /**
         * Reserves the ID for the next object that will be read. This is called only the first time an object is encountered.
         *
         * @param type The type of object that will be read.
         *
         * @return The ID, which is stored more efficiently if it is positive and must not be -1 or -2.
         */
        @Override
        public int nextReadId(Class type)
        {
            int id = new Integer(readObjects.size());
            if (id % 1000 == 0)
                System.out.println("readObject.size=" + id);
            readObjects.add(null);
            return id;
        }

        /**
         * Sets the ID for an object that has been read.
         *
         * @param id The ID from {@link #nextReadId(Class)}.
         */
        @Override
        public void setReadObject(int id, Object object)
        {
            readObjects.set(id, object);
        }

        /**
         * Returns the object for the specified ID. The ID and object are guaranteed to have been previously passed in a call to
         * {@link #setReadObject(int, Object)}.
         */
        @Override
        public Object getReadObject(Class type, int id)
        {
            return readObjects.get(id);
        }

        /**
         * Called by {@link com.esotericsoftware.kryo.Kryo#reset()}.
         */
        @Override
        public void reset()
        {
            writeObjects.clear();
            readObjects.clear();
        }

        /**
         * Returns true if references will be written for the specified type.
         *
         * @param type Will never be a primitive type, but may be a primitive type wrapper.
         */
        @Override
        public boolean useReferences(Class type)
        {
            return !Util.isWrapperClass(type);
        }
    }

You received this message because you are subscribed to a topic in the Google Groups "kryo-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/kryo-users/op81RmHXOsM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to kryo-users+...@googlegroups.com.

[Nate]

On Fri, Aug 16, 2013 at 10:09 PM, Brian Erst <brian...@gmail.com> wrote:

I quickly learned the errors of my ways of not using IdentityHashMap. :D  ("Why the hell is this enum serializing as 1 and deserializing as 3???")

:D

 

I moved back to using my "full size" messages, which end up calling CustomReferenceResolver#addWriteObject about 2.8M times per message and it's just happily churning away.

Good to hear. IdentityHashMap is likely a better choice for a map that is so put-heavy. I'll consider changing the default implementation.

 

If I have time to build a local copy of kryo and play with it, I'll look into why the IdentityObjectIntMap gets so carried away

This would be great, as a lot of people use IdentityObjectIntMap and its variants, not just in Kryo but in other projects (libgdx).

-Nate

[mongonix]

On Thursday, August 15, 2013 8:55:02 PM UTC+2, mongonix wrote:

On Thursday, August 15, 2013 4:55:06 PM UTC+2, Brian Erst wrote:

Has there been any movement on trying to get continuations into trunk? I'm bumping into the same sort of ArrayOutOfBounds issues in IdentiyObjectInMap and/or stack overflows on nested graphs.

I'll provide tomorrow an updated patch for continuations based on the current trunk and containing some bug fixes. But the overall code structure of the patch hasn't changed a lot since I posted the original version of continuations a while ago.

I provided the updated version of my continuations patch based on current trunk. You can find it at: https://code.google.com/p/kryo/issues/detail?id=103

-Leo

[Shengzhe Hu]

Hi Leo, 
Is there any chance that you still have this patch? Cause the link of google code is not avaiable right now.
Really need this solution, it will be a big help!