what is the purpose of __slots__

[krastano...@gmail.com]

Are they used in all the Basic classes for anything other than saving
space? Is the difference in space usage that big?

[krastano...@gmail.com]

Slightly related: Why __new__ is used so often instead of simply
__init__? In basic for example it seams like __init__ is good enough.

[Ronan Lamy]

The only purpose of __slots__ is to save memory, though it doesn't save
a lot. Until PR #1162 gets merged, it's also abused by the pickling
system to give a list of the attributes to save, which makes pickles
unnecessarily large.

Le samedi 05 mai 2012 à 18:39 +0200, krastano...@gmail.com a
écrit :

> Slightly related: Why __new__ is used so often instead of simply
> __init__? In basic for example it seams like __init__ is good enough.
>

There are many cases where __new__ has to be used because the class
constructor can return instances of other classes (cf. Add(1, 2)). It
seems easier to use __new__ all the time than to have to worry about the
interaction of __new__ and __init__.

[Aaron Meurer]

I'm not sure about this. If you try commenting out the empty
__slots__ definition for AtomicExpr, for example, the core tests take
almost twice as long (41 seconds for me vs. 25 in master). It's not
clear why, though. Commenting out the __slots__ in Basic, which is
the only one that I know of that is non-empty, makes the tests take 28
seconds, and commenting out the empty __slots__ for Expr makes them
take 38 seconds.

But I imagine if we were to remove all __slots__ definitions in the
core, that things would be much slower.

Aaron Meurer

A lot of classes in the core have __new__ methods that return
Basic.__new__(cls, *args, **kwargs) (do a grep for Basic\.__new__ to
see what I mean). Any idea why this is done?

Aaron Meurer

[krastano...@gmail.com]

On 5 May 2012 20:56, Aaron Meurer <asme...@gmail.com> wrote:
> I'm not sure about this.  If you try commenting out the empty
> __slots__ definition for AtomicExpr, for example, the core tests take
> almost twice as long (41 seconds for me vs. 25 in master).  It's not
> clear why, though.  Commenting out the __slots__ in Basic, which is
> the only one that I know of that is non-empty, makes the tests take 28
> seconds, and commenting out the empty __slots__ for Expr makes them
> take 38 seconds.
>
> But I imagine if we were to remove all __slots__ definitions in the
> core, that things would be much slower.
>

Almost rhetorical questions follow:
It would be a great research task for Google Code In. But seriously, I
really don't get that. Is a dictionary lookup so expensive in python?
Any ideas about pypy?

>
> A lot of classes in the core have __new__ methods that return
> Basic.__new__(cls, *args, **kwargs) (do a grep for Basic\.__new__ to
> see what I mean).  Any idea why this is done?
>

Maybe it is to have _hashable_content automatically updated with the
stuff in _args and srepr&co automatically aware of the new arguments.
I am not sure...

[krastano...@gmail.com]

>>
>> But I imagine if we were to remove all __slots__ definitions in the
>> core, that things would be much slower.
>>

It looks like it. __slots lookup is faster than __dict__ lookup in
CPython and according to the mailing list of cpython it is an
implementation deficiency.

However it has no effect on pypy (interesting discussion about their
implementation):
http://morepypy.blogspot.fr/2010/11/efficiently-implementing-python-objects.html

[Ronan Lamy]

Le samedi 05 mai 2012 à 12:56 -0600, Aaron Meurer a écrit :
> I'm not sure about this. If you try commenting out the empty
> __slots__ definition for AtomicExpr, for example, the core tests take
> almost twice as long (41 seconds for me vs. 25 in master). It's not
> clear why, though. Commenting out the __slots__ in Basic, which is
> the only one that I know of that is non-empty, makes the tests take 28
> seconds, and commenting out the empty __slots__ for Expr makes them
> take 38 seconds.

When I remove __slots__, I don't see any meaningful difference. Either
something's very different between Mac and Ubuntu, or your timings are
bogus. Considering that they are very erratic, I guess it's the latter.

> But I imagine if we were to remove all __slots__ definitions in the
> core, that things would be much slower.

If a class doesn't have __slots__, __slots__ declarations in its
subclasses have no effect. So if you remove __slots__ from Basic, it's
basically as if you'd removed it from everywhere.

[Aaron Meurer]

So are there any disadvantages of having __slots__? Is pickling
always different when it is defined, or only in the current master?

Aaron Meurer

> --
> You received this message because you are subscribed to the Google Groups "sympy" group.
> To post to this group, send email to sy...@googlegroups.com.
> To unsubscribe from this group, send email to sympy+un...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/sympy?hl=en.
>

[Aaron Meurer]

On Sat, May 5, 2012 at 7:56 PM, Ronan Lamy <ronan...@gmail.com> wrote:
> Le samedi 05 mai 2012 à 12:56 -0600, Aaron Meurer a écrit :
>> I'm not sure about this.  If you try commenting out the empty
>> __slots__ definition for AtomicExpr, for example, the core tests take
>> almost twice as long (41 seconds for me vs. 25 in master).  It's not
>> clear why, though.  Commenting out the __slots__ in Basic, which is
>> the only one that I know of that is non-empty, makes the tests take 28
>> seconds, and commenting out the empty __slots__ for Expr makes them
>> take 38 seconds.
>
> When I remove __slots__, I don't see any meaningful difference. Either
> something's very different between Mac and Ubuntu, or your timings are
> bogus. Considering that they are very erratic, I guess it's the latter.

I think perhaps the test in test_args that searches through all files
may be messing up the timings. So I temporarily removed that file.
Even so, I get variations of up to 20 seconds even in master, so it's
hard to say. I ran the core tests (./bin/test core) in master three
times, and got 29.00, 42.10, and 27.13 seconds. I commented out
__slots__ for AtomicExpr and ran the core tests three times and got
55.91, 48.94, and 39.44. So I don't know. There definitely seems to
be a pattern of no __slots__ being slower. The gross variations on my
computer are probably due to the fact that I have more than a dozen
other programs running at the same time, and the fact that my hard
disk is slow enough to make a huge difference whether the data happens
to be in memory or not.

I wish we had a working benchmark system (and not just a working one,
but one with up-to-date benchmarks). Comparing timings of tests
doesn't seem very helpful. Can you suggest something that might take
more time without __slots__ that I can test?

And as I asked on the other thread, is there a good reason to not have
__slots__? It's just one line, and since they are all empty, it
doesn't really seem to cause any issues. (sorry if I've asked this
before, I forgot the answer)

Aaron Meurer

[krastano...@gmail.com]

>
> I wish we had a working benchmark system (and not just a working one,
> but one with up-to-date benchmarks).  Comparing timings of tests
> doesn't seem very helpful.  Can you suggest something that might take
> more time without __slots__ that I can test?

I can test some of these manually on an idle system. I will upload the
results soon.

>
> And as I asked on the other thread, is there a good reason to not have
> __slots__?  It's just one line, and since they are all empty, it
> doesn't really seem to cause any issues.  (sorry if I've asked this
> before, I forgot the answer)
>

For me it is a pedagogical difficulty. There is something used all
over sympy for no clear reason (to a new user). When you subclass you
do not know if you must follow this style. It is just not "one clear
way to do it".

If the space and time speedups are not substantial it will be
conceptually simpler to remove the slots (and I have heard that the
Jython guys don't like how slots are used in sympy?).

[krastano...@gmail.com]

timing for bin/test core

with slots (15 samples):
measures: sys_time, user_time, max_res_memory
averages: 0.181 s 14.22 s 240.5 KB
std_dev: 0.027 0.59 0.57


slots commented on PickableWithSlots (17 samples):
measures: sys_time, user_time, max_res_memory
averages: 0.172 s 14.02 s 241.1 KB
std_dev: 0.022 0.54 0.68


slots commented on Basic (25 samples):
measures: sys_time, user_time, max_res_memory
averages: 0.188 s 14.76 s 266.6 KB
std_dev: 0.030 0.52 0.50


All this on a KVM virtual machine that reports two cores of
Intel(R) Core(TM)2 Duo CPU T7700 @ 2.40GHz
running debian testing

There is almost no difference.

So after the merge of Ronan's refactoring of the assumption system I
will post a PR with removed slots (just in order to have more people
run test on it).

If it ever gets to the point of being discussed for merging, the advantages are:
- Jython gets happier
- no more wondering "should I define __slots__" or not.

[Ronan Lamy]

Le dimanche 06 mai 2012 à 13:28 +0200, krastano...@gmail.com a
écrit :

More importantly, it will lift the restrictions directly imposed by
__slots__, making it possible to define any attribute, and to use class
attributes as default values.