Morphia iterator for large result set

[sukan]

I'm trying to iterate through 5 millions records collection by using find(query).iterator(). I have also set the batchSize of the query to 100. The iterator runs through 3 million records and crash out with OutOfMemoryError with following exception stack trace

java.lang.OutOfMemoryError: Java heap space

at java.util.HashMap.resize(HashMap.java:462)

at java.util.HashMap.addEntry(HashMap.java:755)

at java.util.HashMap.put(HashMap.java:385)

at com.google.code.morphia.mapping.cache.DefaultEntityCache.notifyExists(DefaultEntityCache.java:40)

at com.google.code.morphia.mapping.cache.DefaultEntityCache.putEntity(DefaultEntityCache.java:81)

at com.google.code.morphia.mapping.Mapper.fromDb(Mapper.java:472)

at com.google.code.morphia.mapping.Mapper.fromDBObject(Mapper.java:267)

at com.google.code.morphia.query.MorphiaIterator.processItem(MorphiaIterator.java:53)

at com.google.code.morphia.query.MorphiaIterator.next(MorphiaIterator.java:48)

From looking into morphia source code I could see that the entity records are cached, and this cache keeps growing which is probably why the exception. I was wondering if there is any way to get a handle on the cache and flush it periodically

[Ben McCann]

Ditto. This stinks. What'd you do to fix it? I might submit a patch to the fork since 10gen has completely abandoned the original.

[James Green]

Nothing seems to use the flush() method:

https://github.com/jmkgreen/morphia/blob/master/morphia/src/main/java/com/github/jmkgreen/morphia/mapping/cache/DefaultEntityCache.java#L83

I took a very brief look this morning. It seems a new DefaultEntityCache is created per Query. There is a HashMap inside to hold Key instances:

https://github.com/jmkgreen/morphia/blob/master/morphia/src/main/java/com/github/jmkgreen/morphia/mapping/cache/DefaultEntityCache.java#L19

Need to understand what problem that entity cache actually solves to move forwards I think. I'll have time later to look and try to follow-up.

--
You received this message because you are subscribed to the Google Groups "Morphia" group.
To unsubscribe from this group and stop receiving emails from it, send an email to morphia+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[shypy]

I'm running into this issue in version 0.105

Looking at a hexdump, the DefaultEntityCache.existenceMap of the MorphiaIterator is massive after iterating through 1 million instances. Is there a way to flush existenceMap?