Possible memory leak in SerializeTranslatorFactory (Objectify)

0 views
Skip to first unread message

Dexter via StackOverflow

unread,
Mar 28, 2016, 11:40:07 AM3/28/16
to google-appengin...@googlegroups.com

I ran across a memory leak when saving a huge number of entities (>10000) which have these 2 fields :

@Unindex
@Serialize(zip = true)
private EnumMultiset eventCounter; //event counter
@Unindex
@Serialize(zip = true)
private EnumMap eventOccurrence; //map to hold which event occurred when

The stacktrace :

Error for /worker-module/task/schedule java.lang.OutOfMemoryError at java.util.zip.Deflater.init(Native Method) at java.util.zip.Deflater.(Deflater.java:180) at java.util.zip.Deflater.(Deflater.java:189) at com.googlecode.objectify.impl.translate.SerializeTranslatorFactory$1.saveValue(SerializeTranslatorFactory.java:70)

Complete stacktrace.

My analysis is that the SerializeTranslatorFactory, creates Deflater instance in saveValue() method and does not call end() or alternatively finalize() on it. I took help from this blog. Hence it eventually goes out of memory. Also, since the default deflater is not used, the onClose() of DeflaterOutputStream does not automatically call .end();

public void close() throws IOException {
    if (!closed) {
        finish();
        if (usesDefaultDeflater) //false
            def.end();
        out.close();
        closed = true;
    }
}


Please DO NOT REPLY directly to this email but go to StackOverflow:
http://stackoverflow.com/questions/36265614/possible-memory-leak-in-serializetranslatorfactory-objectify

Dexter via StackOverflow

unread,
Mar 28, 2016, 12:30:08 PM3/28/16
to google-appengin...@googlegroups.com

I ran across a memory leak when saving a huge number of entities (>10000) which have these 2 fields :

@Unindex
@Serialize(zip = true)
private EnumMultiset eventCounter; //event counter
@Unindex
@Serialize(zip = true)
private EnumMap eventOccurrence; //map to hold which event occurred when

The stacktrace :

Error for /worker-module/task/schedule java.lang.OutOfMemoryError at java.util.zip.Deflater.init(Native Method) at java.util.zip.Deflater.(Deflater.java:180) at java.util.zip.Deflater.(Deflater.java:189) at com.googlecode.objectify.impl.translate.SerializeTranslatorFactory$1.saveValue(SerializeTranslatorFactory.java:70)

Complete stacktrace.

My analysis is that the SerializeTranslatorFactory, creates Deflater instance in saveValue() method and does not call end() or alternatively finalize() on it. I took help from this blog. Hence it eventually goes out of memory. Also, since the default deflater is not used, the onClose() of DeflaterOutputStream does not automatically call .end();

public void close() throws IOException {
    if (!closed) {
        finish();
        if (usesDefaultDeflater) //false
            def.end();
        out.close();
        closed = true;
    }
}

I have also created an issue for this.

Dexter via StackOverflow

unread,
Mar 29, 2016, 10:05:04 AM3/29/16
to google-appengin...@googlegroups.com

I ran across a memory leak when saving a huge number of entities (>10000) which have these 2 fields :

@Unindex
@Serialize(zip = true)
private EnumMultiset<Type> eventCounter; //event counter
@Unindex
@Serialize(zip = true)
private EnumMap<Type, String> eventOccurrence; //map to hold which event occurred when
Reply all
Reply to author
Forward
0 new messages