can't add variables to instances of built-in classes

[Kent Tong]

Hi,

I can add new variables to user-defined classes like:

>>> class Test:
... pass
...
>>> a=Test()
>>> a.x=100

but it doesn't work if the instances belong to a built-in class such as str or list:

>>> a='abc'
>>> a.x=100
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'str' object has no attribute 'x'

What makes this difference?

Thanks in advance!

[Peter Otten]

By default custom classes have a dictionary (called __dict__) to hold these
attributes. If for every string or integer there were such a dict that would
waste a lot of memory. You can subclass if you need it:

>>> class Str(str): pass
...
>>> s = Str("hello")
>>> s.x = 42
>>> s
'hello'
>>> s.x
42

You can also avoid the dict in your own classes by specifiying slots for
allowed attributes:

>>> class Test:
... __slots__ = ("foo", "bar")
...
>>> t = Test()
>>> t.foo = 42
>>> t.baz = "whatever"

Traceback (most recent call last):
File "<stdin>", line 1, in <module>

AttributeError: 'Test' object has no attribute 'baz'

Use this feature sparingly, only when you know that there are going to be
many (millions rather than thousands) of Test instances.

[Wilson Ong]

> Use this feature sparingly, only when you know that there are going to be
> many (millions rather than thousands) of Test instances.

Why use it sparingly? Is it for extensibility? What if I'm pretty sure that my class is going to have exactly these attributes only?

[Kent Tong]

Hi Peter,

Thanks a lot for your excellent explanation!

[Chris Angelico]

"Pretty sure" isn't enough justification for a relatively meagre
saving of memory. Much easier to leave it flexible. It's only once the
memory saving stops being "relatively meagre" (for instance, with the
built-in 'str' type - there are a LOT of strings in a Python program,
since your variable names are stored as strings) that it's worth doing
that.

ChrisA

[Steven D'Aprano]

It's your own code, you can do anything you like, its not like the Python
Police will come and arrest you.

But *best practice* is for instances to have a __dict__, which allows them
to have instance attributes. Python code expects to be able to inspect, and
write to, the instance __dict__, and if you don't have one, you may break
code that doesn't expect it.

But it is not a hard rule, just a strong recommendation. There are
exceptions. Most built-ins don't have __dict__, partly as an optimization,
and partly to allow sharing of builtins across sub-interpreters;

__slots__ was specifically invented so that classes created in Python could
avoid creating a per-instance __dict__, for cases where you expect to have
vast numbers of instances with only a fixed number of attributes. In this
case, having an unused __dict__ can be wasteful.

But note that in recent versions of Python, there is less need to use
__slots__, and a much smaller benefit. The reason is that the instance
dicts can now share storage for their keys.

https://www.python.org/dev/peps/pep-0412/

__slots__ is not obsolete, but 99% of the time you shouldn't bother with it.



--
Steven
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

[Lawrence D’Oliveiro]

On Monday, July 18, 2016 at 3:04:13 AM UTC+12, Steven D'Aprano wrote:

> __slots__ is not obsolete, but 99% of the time you shouldn't bother with it.

I find __slots__ very useful. I have them right through my Qahirah classes <https://github.com/ldo/qahirah>, for example.

[Peter Otten]

When you remove them, does your library stop working? Is there a significant
increase of memory consumption? Or is there something I didn't think of that
makes them useful for you?

[Ethan Furman]

One of the very nice things about Python is being able to fiddle with objects, both
for debugging and for extending. One of the ways that's accomplished is by adding
attributes to instances, but when __slots__ is defined that ability is lost, which
can make using/debugging those types of objects harder.

__slots__ is a memory optimization, and like most optimazations you shouldn't use
it until you know you need it.

--
~Ethan~

[Lawrence D’Oliveiro]

On Monday, July 18, 2016 at 7:39:09 PM UTC+12, Peter Otten wrote:

> Lawrence D’Oliveiro wrote:
>
>> I find __slots__ very useful. I have them right through my Qahirah classes
>> <https://github.com/ldo/qahirah>, for example.
>

> Or is there something I didn't think of that makes them useful for you?

When you have lots of read/write properties, I find __slots__ to be a good idea.

[bream...@gmail.com]

Please explain why, thank you.

Kindest regards.

Mark Lawrence.

[Lawrence D’Oliveiro]

On Tuesday, July 19, 2016 at 11:12:52 AM UTC+12, bream...@gmail.com wrote:
>
> On Monday, July 18, 2016 at 10:48:15 PM UTC+1, Lawrence D’Oliveiro wrote:
>>

>> <https://github.com/ldo/qahirah>

>> When you have lots of read/write properties, I find __slots__ to be a good
>> idea.
>
> Please explain why, thank you.

I was trying something like

ctx.dashes = ((0.1, 0.03, 0.03, 0.03), 0)

and wondering why it wasn’t working...

[bream...@gmail.com]

This makes no sense to me at all. You appear to be trying to create a tuple, which contains a tuple and an integer. You then say it doesn't work, but imply that using __slots__ fixes the problem. So please explain exactly what you were trying to achieve, the exact error you got, with the complete traceback, and how using __slots__ fixed the problem.

Cheers.

Mark Lawrence.

[Lawrence D’Oliveiro]

No traceback. The lines were simply coming out solid, instead of dashed.

[Chris Angelico]

And __slots__ fixed the problem how, exactly? This sounds like the
sort of cargo cult debugging that I'd expect of PHP programmers ("I
put addslashes around everything and now it works, so in future I'll
use addslashes everywhere"), but around here, we're better than that.

ChrisA

[Lawrence D’Oliveiro]

On Wednesday, July 20, 2016 at 6:19:45 PM UTC+12, Chris Angelico wrote:

>
> On Wed, Jul 20, 2016 at 9:58 AM, Lawrence D’Oliveiro wrote:
>>
>> On Wednesday, July 20, 2016 at 9:24:57 AM UTC+12, bream...@gmail.com wrote:
>>>
>>> On Tuesday, July 19, 2016 at 3:54:12 AM UTC+1, Lawrence D’Oliveiro wrote:
>>>>
>>>> On Tuesday, July 19, 2016 at 11:12:52 AM UTC+12, bream...@gmail.com
>>>> wrote:
>>>>>
>>>>> On Monday, July 18, 2016 at 10:48:15 PM UTC+1, Lawrence D’Oliveiro
>>>>> wrote:
>>>>>>
>>>>>> <https://github.com/ldo/qahirah>
>>>>>> When you have lots of read/write properties, I find __slots__ to be a
>>>>>> good idea.
>>>>>
>>>>> Please explain why, thank you.
>>>>
>>>> I was trying something like
>>>>
>>>> ctx.dashes = ((0.1, 0.03, 0.03, 0.03), 0)
>>>>
>>>> and wondering why it wasn’t working...
>>>
>>> This makes no sense to me at all. You appear to be trying to create a
>>> tuple, which contains a tuple and an integer. You then say it doesn't
>>> work, but imply that using __slots__ fixes the problem. So please explain
>>> exactly what you were trying to achieve, the exact error you got, with the
>>> complete traceback, and how using __slots__ fixed the problem.
>>
>> No traceback. The lines were simply coming out solid, instead of dashed.
>
> And __slots__ fixed the problem how, exactly?

The Context attribute that controls the dash settings is called “dash”, not “dashes”.

> This sounds like the sort of cargo cult debugging that I'd expect of PHP
> programmers ("I put addslashes around everything and now it works, so in
> future I'll use addslashes everywhere"), but around here, we're better than
> that.

OK, boss.

[Steven D'Aprano]

On Wednesday 20 July 2016 16:45, Lawrence D’Oliveiro wrote:

>>>>> I was trying something like
>>>>>
>>>>> ctx.dashes = ((0.1, 0.03, 0.03, 0.03), 0)
>>>>>
>>>>> and wondering why it wasn’t working...

And so are we. Since you've already solved the problem, maybe you could
enlighten us?

>>>> This makes no sense to me at all. You appear to be trying to create a
>>>> tuple, which contains a tuple and an integer. You then say it doesn't
>>>> work, but imply that using __slots__ fixes the problem. So please explain
>>>> exactly what you were trying to achieve, the exact error you got, with the
>>>> complete traceback, and how using __slots__ fixed the problem.
>>>
>>> No traceback. The lines were simply coming out solid, instead of dashed.
>>
>> And __slots__ fixed the problem how, exactly?
>
> The Context attribute that controls the dash settings is called “dash”, not
> “dashes”.

Ah, finally, an actually *useful* reply. Honestly Lawrence, couldn't you have
just said this right at the start, instead of having us drag the answer out of
you over multiple email exchanges?

If you had said right at the beginning "I have a habit of mistyping attribute
names, and using __slots__ ensures I get an immediate error" then we would have
understood you. Instead, you waste our time, *and yours*, leading us up the
garden path by implying that you fixed a display bug by changing something
like:

class Context(object):
def __init__(self):
self.dashes = something

to:

class Context(object):
__slots__ = ("dashes",)
def __init__(self):
self.dashes = something

Perhaps now you understand why you gave us the impression of cargo-cult
debugging.




--
Steve

[Peter Otten]

pylint can detect candidates for accidental attribute creation:

$ cat attrib.py
class Context:
def __init__(self):
self.dashes = None

ctx = Context()
ctx.dasehs = ("foo", "bar")
$ pylint attrib | grep dasehs
W: 6, 4: Attribute 'dasehs' defined outside __init__ (attribute-defined-
outside-init)

[Lawrence D’Oliveiro]

On Wednesday, July 20, 2016 at 7:26:36 PM UTC+12, Peter Otten wrote:

> pylint can detect candidates for accidental attribute creation:

And __slots__ will prevent them outright.

[Lawrence D’Oliveiro]

On Tuesday, July 19, 2016 at 9:48:15 AM UTC+12, I wrote:

> When you have lots of read/write properties, I find __slots__ to be a good
> idea.

Let me amend that. When you have *any* read/write properties, I find __slots__ to be a good idea.

[Peter Otten]

And attributes added intentionally.

[Steven D'Aprano]

As well as those added intentionally.

Sometimes I wonder what it is that you see in Python and why you use it.
You're frequently complaining about features of the language. You go out of
your way to write in the style of Java/C/whatever in Python, which means
you get all the disadvantages of writing verbose and inflexible static
Java/C/whatever PLUS the inefficiency of dynamic Python, so the worst of
both worlds.

I genuinely don't see what you get out of using Python. Would you care to
explain? There must be something you like about it, even if it isn't the
ability to add attributes to (nearly) arbitrary objects without declaring
them, or the duck-typing of booleans.

[Lawrence D’Oliveiro]

On Thursday, July 21, 2016 at 12:11:09 AM UTC+12, Steven D'Aprano wrote:

> [long irrelevant rant deleted]

Just because I pointed out what a load of nonsense you were spouting about __slots__, by giving a counterexample of their usefulness? Man, your pride must be hurt...

[Lawrence D’Oliveiro]

You mean, being able to dynamically add new attributes to an object?

Probably not a good idea to mix that with read/write properties...

[bream...@gmail.com]

I hereby request that the moderators take this idiot offline as he's managed to insult two highly respected members of this community in no time at all, those of course being Steven D'Aprano and Peter Otten.

Kindest regards.

Mark Lawrence.

[Chris Angelico]

This is exactly what Steven was talking about. If you can't handle
dynamic read-write attributes (whether they're properties or simple
attributes is orthogonal to this), why even use Python? You can get a
linter to tell you about misspelled attribute names, which has a huge
advantage over __slots__ in that it can give you the information prior
to actually hitting that line of code (prior to even running the
program - and some editors have integrated linters, so you don't even
need to consciously run the linter, you just see the results as you
type). That gives you virtually the same benefit as C/Java style of
coding, where you're forced to declare all your members in the class;
but as an added bonus, you can *dynamically add attributes*! Don't
think that's useful? Well, the other day I wanted to add a wrapper
around the constructor for someone else's object to stash a bit of
extra information into it (to track down a resource leak - turned out
there was an ever-growing collection of objects, because they weren't
being removed from that object), and if I'd been using Java, the
response would have been "no way does that even make sense". Thanks to
dynamic code, I could do what I needed to and track down the leak. If
the author of that object had used __slots__, though, I'd have been
utterly stuck.

Using __slots__ basically takes your object down to the level of a
Java one. That's generally fine for something immutable (you really
don't need to be adding attributes to the integer 42), but even with
immutables, I've sometimes wanted to add hidden attributes; with
mutable objects, it's common enough that you want to leave the option
open unless you have good reason not to. And "my editor sucks" is not
a good reason; nor is "my debugging skills suck".

ChrisA

[Steven D'Aprano]

Why not?


If you're going to make a comment, would you please include some actual
significant information in your post? Dropping obtuse hints and incomplete
answers are not useful, and they give the strong impression that you are
only replying in order to gain a smug sense of superiority rather than to
disseminate knowledge and understanding.

Remember that other people cannot know the thought in your mind, they can
only judge by the actual words you have written, and if those words
standing alone do not explain your thinking, then people will:

(1) fail to understand the information you are trying to pass on;

(2) imagine that you are intentionally trying to be difficult; and

(3) characterise you as an insufferable, pretentious wanker who thinks that
giving mysterious, unintelligible responses makes him out to be ever-so
smarter than everyone else.


Poor quality answer:

"Probably not a good idea to mix that with read/write properties..."

Better:

"Probably not a good idea to mix that with read/write properties, for the
following reasons: [succinct explanation of the reasons why one should
avoid mixing ordinary attributes and properties]. Here's a longer
discussion [link to URL]."

[Peter Otten]

bream...@gmail.com wrote:

> I hereby request that the moderators take this idiot offline as he's

Mark, please behave yourself.

[Lawrence D’Oliveiro]

On Thursday, July 21, 2016 at 2:29:53 PM UTC+12, Chris Angelico wrote:

> Using __slots__ basically takes your object down to the level of a
> Java one.

Is Java like a bogeyman you use to scare people with if they don’t write Python code the way you like it?

[Chris Angelico]

LOL! Not really; it's just that Java's objects are way less featured
than Python's.

ChrisA

[Lawrence D’Oliveiro]

On Thursday, August 4, 2016 at 3:07:17 PM UTC+12, Chris Angelico wrote:

> On Thu, Aug 4, 2016 at 12:34 PM, Lawrence D’Oliveiro wrote:
>> On Thursday, July 21, 2016 at 2:29:53 PM UTC+12, Chris Angelico wrote:
>>
>>> Using __slots__ basically takes your object down to the level of a
>>> Java one.
>>
>> Is Java like a bogeyman you use to scare people with if they don’t write
>> Python code the way you like it?
>
> LOL! Not really; it's just that Java's objects are way less featured
> than Python's.

Using Java as an example of how it can keep you out of trouble where Python won’t is fine; using it as a stick to commit personal attacks on people is not.