OutOfMemoryError on big save

[Salomon Brys]

Hi guys,

I have a real issue, and it is kind of VERY urgent so any help is welcome ;)

Here is the code :

data.save().entities(new TranslatedIterable<EventModel, MessageModel >(data.load().type(MessageModel.class)) {
    @Override
    protected Iterable<EventModel> transform(MessageModel from) {
       EventModel ret =_eventManager.createCommentPost(from);

        int i = num.incrementAndGet();

        if (i % PRINT_EVERY == 0)

            logger.log("Proccessed " + i + " comments...");

        return ret;

    }
}).now();

The thing is, for the next version of my app, I need to create an EventModel for each MessageModel ever posted.

In the app, there are over 5000 MessageModel.

TranslatedIterable<F, T> is a util class that simply gets an Iterable<F> and acts as an Iterable<T> that transforms on demand from F to T when called to Iterator.next();

The problem is I get an OutOfMemoryError after processing 800 to 1000 MessageModel.

What can I do ?

Bien cordialement,

Salomon BRYS

--

     

[Mat Jaggard]

Although not always useful, it might be good to know where the OutOfMemoryException is thrown from.

It seems to me that com.googlecode.objectify.impl.engine.WriteEngine.save might be causing the problem since it iterates through all your entities before saving. I'm sure there's some good reason why Jeff had to do it this way. For really big data work it would be nice to chunk it - not only would this save on memory but then you could be transforming a second chunk of POJOs into Entities whilst the first chunk is saving.

I'm afraid I can't help much more, but if there's any way that you could split your work into chunks (of 100 MessageModels for example), I'm sure that would help. Of course you'll probably then hit another limitation, but you can hope not.

Something like this:

Iterator<EventModel> it = new TranslatedIterable<EventModel, MessageModel >(data.load().type(MessageModel.class)) {

    @Override

    protected Iterable<EventModel> transform(MessageModel from) {

       EventModel ret =_eventManager.createCommentPost(from);

        int i = num.incrementAndGet();

        if (i % PRINT_EVERY == 0)

            logger.log("Proccessed " + i + " comments...");

        return ret;

    }

}.iterator();

while (it.hasNext())

{

List<EventModel> chunk = new ArrayList<EventModel>();

for (int i = 0; i < 100; i++)

{

if (!it.hasNext())

{

break;

}

EventModel em = it.next();

chunk.add(em);

}

data.save().entities(em).now(); //Maybe not now() - could asynchronously complete while reading the next batch??

]

Mat.

[Salomon Brys]

I've already tried to chunk it, but it does not help.

Somehow, Objectify is keeping references of saved and loaded objects somewhere.

Bien cordialement,
Salomon BRYS

--

     

[Mat Jaggard]

When loading data, it's stored in a session cache. As the Javadoc states:

objectify.clear();

Clears the session.  If, for example, you are iterating through large quantities of data you should clear the session after every iteration to prevent memory problems.

[Salomon Brys]

This is what I tried :

I divided the saving by chunk of 50 items, then called data.save().entities(chunk).now() and objectify.clear().

So the session is supposed to be cleared every 50 iterations.

However, the memory keeps rushing up. This have not changed any thing.

Jeff, any idea ?

I really need this fixed... I have a production waiting for this...

Bien cordialement,
Salomon BRYS

--

     

[Jeff Schnitzer]

Bad timing. I'm about to be out of cell coverage for days. Dunno the issue, not obvious from memory. I haven't seen it.

Emergency solution is to throw out ofy instance between iterations. Sorry.

- typed on phone, sorry if curt

[Mat Jaggard]

Do you get the error on the Dev server? If so, can you use a standard java tool (such as jvisualvm) to view which objects are taking the space?

[Salomon Brys]

I don't have enough data on my dev server to get this error :p

And since the bulk loader is broken for java dev server, I'll have to code a way to import data through remote_api manually,

So, I'll do that ;)

Bien cordialement,
Salomon BRYS

--

     

[Salomon Brys]

OMG I found it !!!

It actually was Appsstats !

Using Eclipse MAT (tremendous tool, btw), I saw that the Appsstats filter was actually keeping a copy of each Entity that was loaded or saved.

So I disabled it, and problem solved !

I still need, however, to chunk the data and call objectify.clear() regularly. I'll propose a patch to correct this if I have time.

Anyway Mat, thank you very much for your support, it really helped ;)

Bien cordialement,
Salomon BRYS

--

     

[Mat Jaggard]

Well done! I'll have a look at Eclipse MAT at some point.

Thanks :-)

[jon]

OMG I found it !!!

It actually was Appsstats !

Ah yes, this happened to me too. Sorry it didn't cross my mind when I saw your post.