Can I get the total memory usage of Python?
Haigu
to check for memory leaks. So is there any way to know how much memory is
used by python?
Also is there any way to know how many certain kind of objects are
allocated? say, how many intergers ?
"Martin v. Löwis"
In the release build, you can only find out how many container objects
you have, with gc.getobjects(). In the debug build, you can get a list
of all objects, which you can traverse to count them by type.
Regards,
Martin
Cameron Laird
.
.
What's the feasibility of an enhancement in a future
version of Python to provide greater introspective
power in regard to memory allocation? I'll naively
speculate that, if we can just agree on an interface,
it should impose no particular runtime costs; every-
thing necessary is already available on the C side,
and needs only to be exposed.
--
Cameron Laird <Cam...@Lairds.com>
Business: http://www.Phaseit.net
Personal: http://phaseit.net/claird/home.html
Tim Peters
> What's the feasibility of an enhancement in a future
> version of Python to provide greater introspective
> power in regard to memory allocation? I'll naively
> speculate that, if we can just agree on an interface,
> it should impose no particular runtime costs; every-
> thing necessary is already available on the C side, and needs only to
> be exposed. --
Except that there's no standard way in C to query the platform malloc about
how much memory it's actually using, and all mallocs have hidden overheads.
When allocating a large number of small objects, malloc overhead can be more
than half the story. In 2.3 pymalloc takes most of that out of the platform
malloc's hands (although it still grabs *large* chunks from the platfrom
malloc, and pymalloc still can't know how much overhead the platform malloc
chews up for those).
djw
Isn't that gc.get_objects()?
/d/
Cameron Laird
Right; it's a general principle of common OSs that
t's shockingly hairy to find out what is *really*
going on in memory.
What I meant, although far too elliptically, was just
at the pymalloc level. No, that doesn't correspond
to real memory use, as seen "from the outside"; my own
intentions for memory introspection are satisfied by
anything which reliably enumerates my use of memory,
even if it's inaccurately quantified. It's enough
for me to have a handle on how many and which objects
I'm asking the system to juggle.
Daniel Fackrell
I have begun work on collaborative system that will eventually allow people
to edit and create code for long-lived, persistent objects in a running
server (MUD-type application). For this sort of thing, it would be helpful
to be able to provide the administrators with a general idea of how memory
is being used with respect to time.
One possible use would be to track down plugin modules that are creating
disposable objects, but not destroying all references, leading to a good
old-fashioned leak. Even it can't be isolated to a particular location,
knowing that a leak exists is a good first step toward finding and
eradicating it. A general idea of the rate of leakage is probably a close
second.
--
Daniel Fackrell (newsgrou...@dfackrell.mailshell.com)
When we attempt the impossible, we can experience true growth.
Tim Peters
> Right; it's a general principle of common OSs that
> t's shockingly hairy to find out what is *really*
> going on in memory.
>
> What I meant, although far too elliptically, was just
> at the pymalloc level.
You may have meant that <wink>, but note that pymalloc only intercepts
requests for "small" objects, and only for those small-object subsystems
that explicitly choose to use pymalloc. For example, memory for a large
list comes straight from the platform malloc, and even though ints are small
objects they use their own layer of cruft around the platform malloc. IOW,
there is no choke point for memory allocation in Python above the platform
malloc.
> No, that doesn't correspond to real memory use, as seen "from the
> outside"; my own intentions for memory introspection are satisfied by
> anything which reliably enumerates my use of memory, even if it's
> inaccurately quantified. It's enough for me to have a handle on how
> many and which objects I'm asking the system to juggle.
That's a different question: malloc and pymalloc requests are untyped --
they have no idea what the memory is used for. You may like the stats you
can get in a Python COUNT_ALLOCS build instead: that keeps track of the
current total number of allocations, frees, and highwater mark (max value of
#allocations - #frees over time) on a per-type basis (so, e.g., the stats
for ints are distinct from the stats for floats). That's been available
"forever", but it's a special build because it's not particularly cheap to
keep track of this stuff.
Cameron Laird
Daniel Fackrell <unle...@DELETETHIS.learn2think.org> wrote:
.
.
.
>old-fashioned leak. Even it can't be isolated to a particular location,
>knowing that a leak exists is a good first step toward finding and
>eradicating it. A general idea of the rate of leakage is probably a close
>second.
.
.
Me, too.
Cameron Laird
Tim Peters <tim...@comcast.net> wrote:
.
.
.
>You may have meant that <wink>, but note that pymalloc only intercepts
>requests for "small" objects, and only for those small-object subsystems
>that explicitly choose to use pymalloc. For example, memory for a large
>list comes straight from the platform malloc, and even though ints are small
>objects they use their own layer of cruft around the platform malloc. IOW,
>there is no choke point for memory allocation in Python above the platform
>malloc.
Good summary; thanks.
.
.
.
>That's a different question: malloc and pymalloc requests are untyped --
>they have no idea what the memory is used for. You may like the stats you
>can get in a Python COUNT_ALLOCS build instead: that keeps track of the
>current total number of allocations, frees, and highwater mark (max value of
>#allocations - #frees over time) on a per-type basis (so, e.g., the stats
>for ints are distinct from the stats for floats). That's been available
>"forever", but it's a special build because it's not particularly cheap to
>keep track of this stuff.
>
>
No, indeed. I understand that expense. Hmmmmmm;
I suspect the usual result--the hardest part about
this idea will be to decide what I (we) want. All
the possibilities have unsavory aspects.
Tim Peters
>> That's a different question: malloc and pymalloc requests are
>> untyped -- they have no idea what the memory is used for. You may
>> like the stats you can get in a Python COUNT_ALLOCS build instead:
>> that keeps track of the current total number of allocations, frees,
>> and highwater mark (max value of #allocations - #frees over time) on
>> a per-type basis (so, e.g., the stats for ints are distinct from the
>> stats for floats). That's been available "forever", but it's a
>> special build because it's not particularly cheap to keep track of
>> this stuff.
[Cameron Laird]
> No, indeed. I understand that expense. Hmmmmmm;
> I suspect the usual result--the hardest part about
> this idea will be to decide what I (we) want. All
> the possibilities have unsavory aspects.
They do, but also have favorable aspects. I doubt that thinking about it
will help much before people *try* the various options that already exist,
and see whether they help in practice. COUNT_ALLOCS in particular is
virtually unknown yet very helpful, and does *not* require a debug build too
(COUNT_ALLOCS is orthogonal to debug-versus-release). It allocates extra
memory for type objects only, and there are rarely more than a few hundred
of those, so the memory burden is trivial. The runtime burden is a bit of
extra arithmetic at object creation and destruction times, plus whatever
optimization is lost due to generating extra code at each Py_DECREF site to
increase the per-type "# of frees" count when the refcount hits 0. In all,
it's not a major expense on any count.
In 2.2.2 and 2.3 it's much more helpful than before, because each
user-defined class counts as "a type", so you can get counts of how many
Frobnicator instances are currently alive, the maximum # ever alive, and the
number that have been destroyed. Before 2.2, all instances of all
user-defined classes were of the single InstanceType type, so all the
per-class counts got smushed together.
There's more on this in Misc/SpecialBuilds.txt.