Re: [modwsgi] how to 'real‘ destroy python sub interpreter in mod_wsgi

[Graham Dumpleton]

Why do you have so many applications (as subinterpreters) in the same
process? And what exactly are you trying to do? You may be going about
this in the completely wrong way.

Now, should also be stated that destroying Python sub interpreters
with a process is error prone and can result in memory leaks. This is
partly due to flaws in Python, but also any C extension modules you
may use.

Also, your idea that VmSize will drop when a sub interpreter is
destroyed is somewhat flawed anyway because except for some special
cases, once a process allocates memory from the operating system
level, it doesn't get released back to the operating system and
instead only ends up back on the free list of that process for future
reuse within the same process.

Graham

On 12 September 2012 23:36, pytom <mailli...@gmail.com> wrote:
> hi~ alls,
>
> In mod_wsgi, each of applications has a sub-interpreter, and there're more
> applications with more sub-interpreter, the memory will be exhausted.
>
> So I want to destroy someones with PyDict_DelItemString(wsgi_interpreters,
> $appgrp), it'll invoke Interpreter_dealloc to end sub-interpreter.
>
> But unfortunately, the Memory Used is not slow down. I found 'VmSize' was
> same as before in /proc/$pid/status.
>
> Someone could help me, Thanks a lot~
>
> --
> You received this message because you are subscribed to the Google Groups
> "modwsgi" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/modwsgi/-/t48gfUNoPmEJ.
> To post to this group, send email to mod...@googlegroups.com.
> To unsubscribe from this group, send email to
> modwsgi+u...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/modwsgi?hl=en.

[pytom]

Thanks for your reply.  Now I've two questions,

1. as you said, 'memory leaks' , could it be released with GC module？

2. about 'VmSize',  in my test program, I printf it firstly, then re-printf it after Function 'free', it exactly reduces. 

With destroying sub-interpreter, I also found the func 'free' is invoked several times ( i use gdb attch $pid), 

but Why the ‘VmSize’ is constant？

在 2012年9月15日星期六UTC+8下午1时27分34秒，Graham Dumpleton写道：

[Graham Dumpleton]

On 15 September 2012 00:47, pytom <mailli...@gmail.com> wrote:
> Thanks for your reply. Now I've two questions,
>
> 1. as you said, 'memory leaks' , could it be released with GC module？

Memory leaks in what? They aren't memory leaks if freeing stuff fixes them.

Sounds more like you are creating object loops that garbage collector
cannot break, which destroying a sub interpreter will not fix, or
leaving objects in object caches and not clearing them properly when
they are no longer required to be retained. The latter would be
cleaned up by destroying a sub interpreter, but is the wrong way of
going about it, fix your application design instead.

Overall it sounds to me like you have no idea where the wastage of
memory is coming from and thinking destroying sub interpreters is the
answer, which is the wrong way of going about it.

You should perhaps try and understand where memory is being used
better in your application. Try heapy.

http://guppy-pe.sourceforge.net/

> 2. about 'VmSize', in my test program, I printf it firstly, then re-printf
> it after Function 'free', it exactly reduces.
> With destroying sub-interpreter, I also found the func 'free' is invoked
> several times ( i use gdb attch $pid),
> but Why the ‘VmSize’ is constant？

Are you talking about a C test program. How memory use works in it is
going to be somewhat different to how Python works.

If anything, what you are likely encountering is the one optimisation
in some malloc libraries whereby if you malloc memory and it causes
the operating system memory allocator to allocate more memory pages,
and then you free it straight away and the top memory pages all become
unused again, because it is at the top of available memory, it can
release them back. In Python this can technically happen, but is less
likely because of much greater use of memory allocator for small
objects.

So, do you actually understand properly or not what is causing the
memory usage?

Graham

> https://groups.google.com/d/msg/modwsgi/-/Pbpcmq8gcPUJ.