Steadily increasing memory usage for a long running script using mongoengine

[TTimo]

Hello,

I have a script that used to run for weeks on end with a stable memory consumption pattern. At the time it was only dealing with the Django ORM. We have added MongoDB to our production environment and added a few queries that use mongoengine to the script.

I am now seeing the memory consumption steadily rise, and I'm not finding anything in our local code that would cause that, so I'm thinking it's in the mongoengine module. I am closing the MongoDB connection between each loop of the script, but the memory usage is still rising.

Any suggestions on what might be causing this? I am trying to run this under heap profilers (heapy), but not having much success so far.

Thanks!

TTimo

[Ross Lawley]

Hi,

What version of mongoengine are you using?  What version of pymongo and on what os?  What level of memory consumption are you seeing?  Can you provide an example of the model definitions and the loop?

On a side note you probably don't need to close the connection pool between each loop that will just add to the cost.

Ross

[TTimo]

Ubuntu 10.04.3 LTS

PyMongo 2.3

MongoEngine 0.7.5

The script sleeps 10 minutes between each loop, went from 22M of heap memory to 115M over a 4h period. Steadily adding with each loop.

The operations are nothing special, a few .filter() calls on collections that size up to a few hundreds of thousand of items.

Most of that memory growth is in str types, and that's about as far as my investigation has gone so far as I'm trying to figure out how to use heapy or some other tools to track things down deeper.

TTimo