unbound memory problem

[Amirouche Boubekki]

Using sophy bindings and sophia 2.1 I try to load a several Gb file into a database but even after settings the memory limit the consumption grows without limit.

What I am doing wrong?

Here's more or less the code I use:

            from sophy import Sophia
            from msgpack import loads, dumps

            env = Sophia(db, auto_open=False)
            env.memory_limit = 1024 * 1024 * 1024
            env.open()

            posts = env.create_database('Post', 'u32')

            filepath = os.path.join('dump', 'Posts.xml')
            for data in iterate(filepath):
                 posts[int(data['Id'])] = dumps(data)
            env.close()

Thanks!

[Dmitry Simonenko]

Hi,

It is possible to expect that memory usage will be memory_limit * 2 in some cases, due to malloc() behaviour (fragmentation).

Sophia implements memory limit only on logical level. Does the memory consumption stops at some point?

Thanks

[Amirouche Boubekki]

On Saturday, July 30, 2016 at 2:16:47 PM UTC+2, Dmitry Simonenko wrote:

Hi,

It is possible to expect that memory usage will be memory_limit * 2 in some cases, due to malloc() behaviour (fragmentation).

Sophia implements memory limit only on logical level. Does the memory consumption stops at some point?

no it doesn't. It goes well above 2Giga of RAM until there is no more memory left and it crash

[Charles Leifer]

From what I could tell when investigating your issue, the setting is working correctly.

If, after setting it and opening the env, I did "print env.memory_limit" and got back the value I set.

[Dmitry Simonenko]

Does "print env.memory_limit" equal to printing the value obtained from sp_getstring(env, "memory.limit", 0) ?

[Charles Leifer]

Actually it uses sp_getint(env, "memory.limit"). If you look at the source code for sophy you can see that accessing the memory limit revolves around this function:

https://github.com/coleifer/sophy/blob/2cac2f5ddf4ba8ac4e3ab007b857d539baf73ebc/sophy.pyx#L83-L91

When the memory limit property is defined on the Sophy class, it indicates that the value is an integer.

[Amirouche Boubekki]

The memory leak was in my code. I'm very sorry about the confusion.

[Dmitry Simonenko]

Ok, good that the problem has been resolved. Thanks for telling.