New modules and packages
Ludwig Hähne
happy new year to both of you. I'm almost through with the changes I
have planned to make in my branch, only the docs are not yet consistent.
Some of the changes may need a bit of discussion before they can be
merged back to the trunk:
* The heapmonitor "core" has been moved to a new module
'pympler.tracker.classes'. The module-level functions have been packed
into the new ClassTracker class. The new name shall already indicate
what can be tracked by this module; and allow other tracking facilities
be placed alongside.
* The presentation logic (text/html-output) have been moved out into the
pympler.tracker.mstats module. Maybe this module should be moved to the
pympler.gui package? Furthermore, mstats might not be the best name ...
* There is a new pympler.gui package. So far, only non-interactive,
static visualization is therein - therefore it could also be called
pympler.visual. Depending on what we plan to put there in the future,
one or the other makes more sense.
* The garbage visualization (as a graph) has got its own module
pympler.gui.garbage and inherits from a generic graph visualization
module pympler.gui.graph. I first tried to inherit from the muppy
ReferenceBrowser (therefrom the name GraphBrowser/GarbageBrowser).
However, there turned out to be few resemblances between the reference
browsing and graph browsing implementations. Reference browsers start
with a root object and recursively dive into the referents, whereas
graph browsers illustrate a (fixed) collection of objects. Apart from
that, the GarbageBrowser implementation has been improved considerably.
Edge labeling now works much better, and graphs can be directly rendered
using graphviz.
* There is a new package/module pympler.util.stringutils. General
utility functions for truncating and "pretty-printing" have been placed
there.
* Already applied to trunk: The new pympler.process module for
collecting process-level memory information from the operating system.
Thanks for any input.
Ludwig
Robert Schuppenies
Up front, I really like the new ClassTracker API. Clean, good
documentation, looks great.
I have inlined some comments I have from reviewing the branch.
Ludwig Hähne wrote:
> I'm almost through with the changes I
> have planned to make in my branch, only the docs are not yet consistent.
In the docs you mention SCons without any reference or explanation.
To the unaware reader this could be confusing.
Also, I found references to both "asizeof" and "sizer". Maybe this
release is a good point to fix that package name to either one.
Not sure if this is a bug or not: You write
"tracker.track_class(MyObject, ...)", but shouldn't it be "MyClass"
you pass to the method?
> Some of the changes may need a bit of discussion before they can be
> merged back to the trunk:
>
> * The heapmonitor "core" has been moved to a new module
> 'pympler.tracker.classes'. The module-level functions have been packed
> into the new ClassTracker class. The new name shall already indicate
> what can be tracked by this module; and allow other tracking facilities
> be placed alongside.
I agree with the tracker package. About the ClassTracker: it also
tracks individual objects, doesn't it? And in the docs you refer to
it both as "Class Tracker" and "ClassTracker". I would like it to be
either one or the other.
We should think about whether it may be more convinient to change
the import from "pympler.tracker.classes import ClassTracker" to
"from pympler.tracker import ClassTracker"
> * The presentation logic (text/html-output) have been moved out into the
> pympler.tracker.mstats module. Maybe this module should be moved to the
> pympler.gui package?
Maybe. Let's see if we can reuse code from it for both muppy and sizer.
> Furthermore, mstats might not be the best name ...
What about just stats? Or will be have statistics on anything else
than memory?
> * There is a new pympler.gui package. So far, only non-interactive,
> static visualization is therein - therefore it could also be called
> pympler.visual. Depending on what we plan to put there in the future,
> one or the other makes more sense.
I think there will be other gui stuff, too, so better leave it "gui".
> * The garbage visualization (as a graph) has got its own module
> pympler.gui.garbage and inherits from a generic graph visualization
> module pympler.gui.graph. I first tried to inherit from the muppy
> ReferenceBrowser (therefrom the name GraphBrowser/GarbageBrowser).
> However, there turned out to be few resemblances between the reference
> browsing and graph browsing implementations. Reference browsers start
> with a root object and recursively dive into the referents, whereas
> graph browsers illustrate a (fixed) collection of objects. Apart from
> that, the GarbageBrowser implementation has been improved considerably.
> Edge labeling now works much better, and graphs can be directly rendered
> using graphviz.
That's fine. The ReferenceBrowser was a quick, rather than a good
solution :)
I was thinking of whether GraphBrowser is a good name, because does
it allow to browse the the graph (or just looking at it)?
> * There is a new package/module pympler.util.stringutils. General
> utility functions for truncating and "pretty-printing" have been placed
> there.
That'll be useful. muppy needs these things too.
> * Already applied to trunk: The new pympler.process module for
> collecting process-level memory information from the operating system.
Do we want to put any restrictions on top-level modules? Currently
we have 7 and rising. What about putting it into the utils package.
stringutils may feel lonely there :) Or we declare 'process' an
explizit feature of pympler and should promote it as such.
cheers,
robert
Ludwig Hähne
> Hi Ludwig.
>
> Up front, I really like the new ClassTracker API. Clean, good
> documentation, looks great.
> I have inlined some comments I have from reviewing the branch.
Thanks for looking into it. Some comments below.
> Ludwig Hähne wrote:
>> I'm almost through with the changes I
>> have planned to make in my branch, only the docs are not yet consistent.
>
> In the docs you mention SCons without any reference or explanation.
> To the unaware reader this could be confusing.
Thanks for the pointer. This is old stuff inherited from the GSoC Wiki
doc. No need to mention SCons in this context.
> Also, I found references to both "asizeof" and "sizer". Maybe this
> release is a good point to fix that package name to either one.
+1
I think we already agreed to use the new name "sizer" after the first
release.
> Not sure if this is a bug or not: You write
> "tracker.track_class(MyObject, ...)", but shouldn't it be "MyClass"
> you pass to the method?
This could be misleading, indeed.
>> Some of the changes may need a bit of discussion before they can be
>> merged back to the trunk:
>>
>> * The heapmonitor "core" has been moved to a new module
>> 'pympler.tracker.classes'. The module-level functions have been packed
>> into the new ClassTracker class. The new name shall already indicate
>> what can be tracked by this module; and allow other tracking facilities
>> be placed alongside.
>
> I agree with the tracker package. About the ClassTracker: it also
> tracks individual objects, doesn't it? And in the docs you refer to
Actually, the correct term would be InstanceTracker - because it is
tracking class instances and not class objects, but I think
InstanceTracker does not appeal. You are right: One can track individual
objects using track_object, but it's somewhat pointless in practice.
I've merely exposed and documented it so that track_class is easier to
understand.
> it both as "Class Tracker" and "ClassTracker". I would like it to be
> either one or the other.
Fixed (hopefully).
> We should think about whether it may be more convinient to change
> the import from "pympler.tracker.classes import ClassTracker" to
> "from pympler.tracker import ClassTracker"
I agree. I will leave it as it is for now, though.
>> * The presentation logic (text/html-output) have been moved out into the
>> pympler.tracker.mstats module. Maybe this module should be moved to the
>> pympler.gui package?
>
> Maybe. Let's see if we can reuse code from it for both muppy and sizer.
Probably not (in the near future). The mstats classes are tightly
coupled to the type of data gathered by the ClassTracker. I doubt there
are many things interesting for muppy or sizer.
>> Furthermore, mstats might not be the best name ...
>
> What about just stats? Or will be have statistics on anything else
> than memory?
stats works for me.
>> * There is a new pympler.gui package. So far, only non-interactive,
>> static visualization is therein - therefore it could also be called
>> pympler.visual. Depending on what we plan to put there in the future,
>> one or the other makes more sense.
>
> I think there will be other gui stuff, too, so better leave it "gui".
>
>> * The garbage visualization (as a graph) has got its own module
>> pympler.gui.garbage and inherits from a generic graph visualization
>> module pympler.gui.graph. [...]
>
> That's fine. The ReferenceBrowser was a quick, rather than a good
> solution :)
> I was thinking of whether GraphBrowser is a good name, because does
> it allow to browse the the graph (or just looking at it)?
No, it is just for visualization. What about ReferenceGraph and
GarbageGraph?
>> * There is a new package/module pympler.util.stringutils. General
>> utility functions for truncating and "pretty-printing" have been placed
>> there.
>
> That'll be useful. muppy needs these things too.
>
>> * Already applied to trunk: The new pympler.process module for
>> collecting process-level memory information from the operating system.
>
> Do we want to put any restrictions on top-level modules? Currently
> we have 7 and rising. What about putting it into the utils package.
> stringutils may feel lonely there :) Or we declare 'process' an
> explizit feature of pympler and should promote it as such.
I would prefer the latter. ATM, the module is not too powerful but it
closes a gap in the pympler memory profiling API (sizing the whole
process). There is also some third-party Python code in the web for
parsing smaps and friends. We might want to add such stuff later.
Ludwig
PS: I have also included a newly written tutorial which profiles a SCons
patch by Jean ;)