[Python-ideas] Add a dict with the attribute access capability

[Serhiy Storchaka]

In 3.7 I have removed an old-deprecated plistlib.Dict. [1] Actually it
already was deprecated when the plistlib module was added to the regular
stdlib in Python 2.6.

This is a dict subclass which allows to access items as attributes.

d = plistlib.Dict()
d['a'] = 1
assert d.a == 1
d.b = 2
assert d['b'] == 2

Raymond noticed that that capability seemed nice to have.

What do you think about reviving this type as general purpose type in
collections or types? Perhaps it can be convenient for working with
JSON, plists, configuration files, databases and in other cases that
need a dict with string keys.

If reintroduce it, there are open questions.

1. The name of the type.

2. The location of the type. collections or types? Or other variants?

3. How it will collaborate with OrderedDict, defaultdict, etc?

4. Should it be a dict subclass, or a mixin, or a proxy? Or add several
types?


[1] https://bugs.python.org/issue29196

_______________________________________________
Python-ideas mailing list
Python...@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Daniel Moisset]

As reference of prior art, there is https://pypi.python.org/pypi/munch in PyPI

--

Daniel F. Moisset - UK Country Manager - Machinalis Limited

www.machinalis.co.uk

Skype: @dmoisset T: + 44 7398 827139

1 Fore St, London, EC2Y 9DT

Machinalis Limited is a company registered in England and Wales. Registered number: 10574987.

[Nick Timkovich]

On Wed, Nov 29, 2017 at 7:21 AM, Daniel Moisset <dmoi...@machinalis.com> wrote:

As reference of prior art, there is https://pypi.python.org/pypi/munch in PyPI

Box is also popular as it permits deeper nesting: https://pypi.python.org/pypi/python-box/ https://github.com/cdgriffith/Box  Addict is similar: https://pypi.python.org/pypi/addict/2.1.1 https://github.com/mewwts/addict and I've seen AttrDict a few times in code I maintain: https://pypi.python.org/pypi/attrdict https://github.com/bcj/AttrDict 

With a cursory search, also found:

* DotMap: https://pypi.python.org/pypi/dotmap https://github.com/drgrib/dotmap

* EasyDict https://pypi.python.org/pypi/easydict https://github.com/makinacorpus/easydict

* Treedict (older) https://pypi.python.org/pypi/treedict

* Bunch (older) https://pypi.python.org/pypi/bunch

...I spy a theme :P

Haven't dug into them that much, but I'd try to answer most questions with how they do it. I'm not sure if the recursive 'items as attributes within items as attributes...' feature is a can of worms or if it's something like defaultdict(dict), but with more levels. Getting deeply nested items with dots seems to be the usual use (i.e. complicated JSONs), and I think the degree of laziness in creating/setting nested items is variable.

Nick

[Barry Warsaw]

Serhiy Storchaka wrote:
> In 3.7 I have removed an old-deprecated plistlib.Dict. [1] Actually it
> already was deprecated when the plistlib module was added to the regular
> stdlib in Python 2.6.
>

> Raymond noticed that that capability seemed nice to have.

So nice in fact that I'm sure I've reimplemented something similar
several times. :)

> What do you think about reviving this type as general purpose type in
> collections or types? Perhaps it can be convenient for working with
> JSON, plists, configuration files, databases and in other cases that
> need a dict with string keys.
>
> If reintroduce it, there are open questions.
>
> 1. The name of the type.
>
> 2. The location of the type. collections or types? Or other variants?
>
> 3. How it will collaborate with OrderedDict, defaultdict, etc?
>
> 4. Should it be a dict subclass, or a mixin, or a proxy? Or add several
> types?

I also wonder whether PEP 557 dataclasses could provide this except in
the opposite direction, e.g. by optionally adding __getitem__ support.

-Barry

[Ivan Levkivskyi]

On 29 November 2017 at 20:11, Barry Warsaw <ba...@python.org> wrote:

Serhiy Storchaka wrote:
> In 3.7 I have removed an old-deprecated plistlib.Dict. [1] Actually it
> already was deprecated when the plistlib module was added to the regular
> stdlib in Python 2.6.
>
> Raymond noticed that that capability seemed nice to have.

So nice in fact that I'm sure I've reimplemented something similar
several times. :)

> What do you think about reviving this type as general purpose type in
> collections or types? Perhaps it can be convenient for working with
> JSON, plists, configuration files, databases and in other cases that
> need a dict with string keys.
>
> If reintroduce it, there are open questions.
>
> 1. The name of the type.
>
> 2. The location of the type. collections or types? Or other variants?
>
> 3. How it will collaborate with OrderedDict, defaultdict, etc?
>
> 4. Should it be a dict subclass, or a mixin, or a proxy? Or add several
> types?

I also wonder whether PEP 557 dataclasses could provide this except in
the opposite direction, e.g. by optionally adding __getitem__ support.

This was discussed in https://github.com/ericvsmith/dataclasses/issues/21

and it was decided to postpone this.

--

Ivan

[Chris Barker]

On Wed, Nov 29, 2017 at 12:52 AM, Serhiy Storchaka <stor...@gmail.com> wrote:

This is a dict subclass which allows to access items as attributes.

d = plistlib.Dict()
d['a'] = 1
assert d.a == 1
d.b = 2
assert d['b'] == 2

 

What do you think about reviving this type as general purpose type in collections or types

Am I I the only one that thinks this is a "Bad Idea"?

For me, it simply confuses even more the question of "is this code or is this data?" -- which is a difficult enough design question in a dynamic language.

And the couple of libraries I"ve worked with that do this I liked at first, but grew to find problematic.

The key problem is that keys in dicts can be anything immutable, while names have a lot more restrictions on them. And in a case like this, you can't use a key that happens to be the same as an existing dict attribute -- at least not in the same way.

In particular, I've struggled with the netCDF4 lib and Colander.

I've used netCDF4 more, so I'll talk about that:

netCDF4 is a Python wrapper around the netcdf scientific data file format. netcdf has "variables" whiuch hold arrays of data, and attributes (for meta data).

The netCDF4 python lib wraps a native netcdf4 variable with a python object that python attributes (names) for each of the netcdf variable attributes. Pretty nifty, really, you can do things like:

vel = dataset.variables['velocity']

print("velocity is in units of:", vel.units)

kinda cool, but in fact it gets harder to deal with in "real code" as opposed to quicky scripts than if variables has a dict-like attribute:

print("velocity is in units of:", vel.atts['units'])

Why? because netcdf variables can have arbitrary attributes, so your code tends to be hard-coded for specific data sets, and breaks fast if the data files are not compliant.

That's what I mean by the difference between code and data -- the attributes of a netcdf variable is data -- not code.

Granted, with Python, you can use try: except AttributeError, and setattr, and all that to work with these attributes as though they are data, but then you lose all the advantages of having them as attributes anyway. setattr and the like shulud be used for when metaprogramign is called for - not for everyday data manipulation.

Oh, and it's also uglier to get/set values in bulk (though the proposed dict_with_attributes would present both APIs, so not as restrictive.)

You also have problems with keys that mirror actual attributes. So you use the dict interface to deal with those -- but if you have to choose your interface according to what the values of the data are -- you've got the whole, it's not code, it's data! problem again.

Anyway -- those are my $0.02 -- from experience working with such things in the real world, with real code, and real data....

-Chris

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris....@noaa.gov

[Ethan Furman]

On 11/29/2017 03:34 PM, Chris Barker wrote:

> Am I I the only one that thinks this is a "Bad Idea"?

No, you're not.

> For me, it simply confuses even more the question of "is this code or is this data?" -- which is a difficult enough
> design question in a dynamic language.

As one of those who have created their own version of AttrDict, I can say it's both neat and tricky to use. I'm happy
to have it* as a third-party package, but I don't think it belongs in the stdlib.

--
~Ethan~


* attribute-access dicts in general, not mine in particular

[Rob Cliffe]

On 29/11/2017 23:34, Chris Barker wrote:

On Wed, Nov 29, 2017 at 12:52 AM, Serhiy Storchaka <stor...@gmail.com> wrote:

This is a dict subclass which allows to access items as attributes.

d = plistlib.Dict()
d['a'] = 1
assert d.a == 1
d.b = 2
assert d['b'] == 2

 

What do you think about reviving this type as general purpose type in collections or types

Am I I the only one that thinks this is a "Bad Idea"?

For me, it simply confuses even more the question of "is this code or is this data?" -- which is a difficult enough design question in a dynamic language.

And the couple of libraries I"ve worked with that do this I liked at first, but grew to find problematic.

I have also implemented something like this (as a convenience, to hold a row from an SQL table).  But it never occurred to me that there should be something like it in the stdlib.
Rob Cliffe

[Steven D'Aprano]

On Wed, Nov 29, 2017 at 10:52:51AM +0200, Serhiy Storchaka wrote:

> In 3.7 I have removed an old-deprecated plistlib.Dict. [1] Actually it
> already was deprecated when the plistlib module was added to the regular
> stdlib in Python 2.6.
>
> This is a dict subclass which allows to access items as attributes.

Isn't that a trivial subclass?

class AttrDict(dict):
def __init__(self, *args, **kw):
super().__init__(*args, **kw)
self.__dict__ = self

Possibly apart from a nicer repr, have I missed anything?

> Raymond noticed that that capability seemed nice to have.
>
> What do you think about reviving this type as general purpose type in
> collections or types? Perhaps it can be convenient for working with
> JSON, plists, configuration files, databases and in other cases that
> need a dict with string keys.

I doubt that it is as convenient as you think. Most of the time I work
with external data, I don't know the keys in advance, so I couldn't use
obj.attr syntax. Configuration files (and equivalent) are possibly an
exception.

But in general I'm suspicious of using attribute access for key lookups.
I cannot help feeling that, while laziness and hubris are virtues in
programmers, this is *too* lazy. This conflates two distinct concepts,
attributes and key/value pairs. Attributes are conceptually part of the
object, as in dog.tail or car.engine, or data about the object, like
page.top_margin or engine.capacity. Key/value data is not.

Another way to look at this would be to say that deleting attributes
is rare and unusual, while deleting keys is generally frequent and
commonplace.

Its not a completely hard distinction, I acknowledge that its already a
bit fuzzy, but I'm not sure it is a good idea to make it even fuzzier.

Consequently I'm strongly opposed to making this standard behaviour for
dict and other mappings, but I'm neutral about making it opt-in for a
distinct mapping type.

> If reintroduce it, there are open questions.
>
> 1. The name of the type.

AttrDict seems like the obvious name.

> 2. The location of the type. collections or types? Or other variants?

It is a mapping, which is a collection, so collections.

> 3. How it will collaborate with OrderedDict, defaultdict, etc?

Does it need to?

> 4. Should it be a dict subclass, or a mixin, or a proxy? Or add several
> types?

Aren't these implementation details?



--
Steve

[Ivan Pozdeev via Python-ideas]

I use argparse.Namespace whenever I need this.

In reply to Chris Barker's concern of "is this code or is this data",
the last time I used it is when I needed to hold an external function
reference in an object instance (if I assigned it to an attribute, it
was converted into an instance method). It just didn't feel right to
invoke it via a container lookup.

A telling name for the object also clears doubts about what it's
supposed to hold.

--
Regards,
Ivan

[Greg Ewing]

Ivan Pozdeev via Python-ideas wrote:
> I needed to hold an external function
> reference in an object instance (if I assigned it to an attribute, it
> was converted into an instance method).

No, that only happens to functions stored in *class* attributes,
not instance attributes.

>>> class A:
... pass
...
>>> a = A()
>>>
>>> def f():
... print("I'm just a function")
...
>>> a.x = f
>>> a.x()
I'm just a function

--
Greg

[Nick Coghlan]

On 30 November 2017 at 05:11, Barry Warsaw <ba...@python.org> wrote:
> Serhiy Storchaka wrote:
>> In 3.7 I have removed an old-deprecated plistlib.Dict. [1] Actually it
>> already was deprecated when the plistlib module was added to the regular
>> stdlib in Python 2.6.
>>
>> Raymond noticed that that capability seemed nice to have.
>
> So nice in fact that I'm sure I've reimplemented something similar
> several times. :)

Note that we do offer a simple namespace type:

>>> from types import SimpleNamespace as ns
>>> data = ns(a=1, b=2, c=3)
>>> data.a
1
>>> vars(data)["a"]
1
>>> vars(data)["a"] = 3
>>> data.a
3

It was added as part of adding sys.implementation, since it's the
storage type used for that:

>>> import sys
>>> type(sys.implementation)
<class 'types.SimpleNamespace'>

So the only thing we don't currently offer is a type that provides
both attribute access and mapping access on the *same* object - for
SimpleNamespace we require that you request the mapping API with
"vars(ns)".

Cheers,
Nick.

--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia

[Ivan Pozdeev via Python-ideas]

On 01.12.2017 1:19, Greg Ewing wrote:
> Ivan Pozdeev via Python-ideas wrote:
>> I needed to hold an external function reference in an object instance
>> (if I assigned it to an attribute, it was converted into an instance
>> method).
>
> No, that only happens to functions stored in *class* attributes,
> not instance attributes.
>
> >>> class A:
> ...    pass
> ...
> >>> a = A()
> >>>
> >>> def f():
> ...    print("I'm just a function")
> ...
> >>> a.x = f
> >>> a.x()
> I'm just a function
>

Well, yes, that was a singleton class, so I kept data in the class
object. Now I can simplify the code by only keeping the instance
reference in the class, thank you. (Without knowing this, that bore no
visible benefits.)

--
Regards,
Ivan