[Python-ideas] Add "default" kw argument to operator.itemgetter and operator.attrgetter

[Vincent Maillol]

Hi everybody,

Our PEP idea would be to purpose to add a global default value for
itemgeet and attrgetter method.

This was inspired from bug 14384 (https://bugs.python.org/issue14384);
opened by Miki TEBEKA.

For example, we could do:

p1 = {'x': 43; 'y': 55}
x, y, z = itemgetter('x', 'y', 'z', default=0)(values)
print(x, y, z)
43, 55, 0

instead of:

values = {'x': 43; 'y': 55}
x = values.get('x', 0)
y = values.get('y', 0)
z = values.get('z', 0)
print(x, y, z)
43, 55, 0

The goal is to have have concise code and improve consistency with
getattr, attrgetter and itemgetter

What are you thinking about this?

MAILLOL Vincent
GALODE Alexandre
_______________________________________________
Python-ideas mailing list
Python...@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Steven D'Aprano]

On Wed, May 02, 2018 at 10:08:55AM +0200, Vincent Maillol wrote:
> Hi everybody,
>
> Our PEP idea would be to purpose to add a global default value for
> itemgeet and attrgetter method.

I'm sorry, I'm not sure I understand what you mean by a global default.
My interpretation of that would be the ability to set a global variable
which acts as default for *all* calls to itemgetter.

itemdefault = "spam"
f = itemgetter(2, 99)
f('abcd')
# => returns ('c', 'spam')

If that's what you mean, I hate it :-)


[...]

> For example, we could do:
>
> p1 = {'x': 43; 'y': 55}
> x, y, z = itemgetter('x', 'y', 'z', default=0)(values)
> print(x, y, z)
> 43, 55, 0

I wouldn't call that a *global* default. If the default has to be
specified on each call, I think that's both acceptible and desirable.

Bug 14384 (https://bugs.python.org/issue14384) suggests that it might be
more desirable to specify the default when you call the getter function,
not when you call the factory:

# this
f = itemgetter(2, 99)
f('abcd', default='spam')
# => returns ('c', 'spam')


# rather than this
f = itemgetter(2, 99, default='spam')
f('abcd')
# => returns ('c', 'spam')


I could go either way. In fact, at the risk of making a more complicated
API, I'm almost inclined to allow both forms...



--
Steve

[Raymond Hettinger]

> On May 2, 2018, at 1:08 AM, Vincent Maillol <vincent...@gmail.com> wrote:
>
> Our PEP idea would be to purpose to add a global default value for
> itemgeet and attrgetter method.

My preference is to not grow that API further. It is creep well beyond its intended uses. At some point, we're really better off just using a lambda.


Raymond

[Steven D'Aprano]

On Wed, May 02, 2018 at 10:28:33PM -0700, Raymond Hettinger wrote:
>
>
> > On May 2, 2018, at 1:08 AM, Vincent Maillol <vincent...@gmail.com> wrote:
> >
> > Our PEP idea would be to purpose to add a global default value for
> > itemgeet and attrgetter method.
>
> My preference is to not grow that API further. It is creep well
> beyond its intended uses.

Intended by whom?

I think you are being too dismissive of actual use-cases requested by
actual users. Default values might not have been the primary use
considered when the API was first invented, but the fact that people
keep asking for this feature should tell us that at least some people
have intended uses that are remaining unmet.

> At some point, we're really better off just using a lambda.

Maybe I'm slow today, but I'm having trouble seeing how to write this as
a lambda.

The single index case would be simple using the proposed binding-
expression from PEP 572:

# replace INDEX and DEFAULT with the values you want
lambda seq: s[0] if (s := seq[INDEX:INDEX+1]) else DEFAULT

but without the binding-expression, I have no idea how to do it cleanly
and correctly in a lambda.

For what it is worth, on my computer, itemgetter with a single index is
20% faster than a lambda using seq[INDEX] (where INDEX is hard-coded in
the lambda), and 60% faster than a lambda using seq[INDEX:INDEX+1][0].

According to the bug report linked to earlier, one of the intended uses
for itemgetter is when performance matters.


--
Steve

[Serhiy Storchaka]

03.05.18 08:28, Raymond Hettinger пише:

>> On May 2, 2018, at 1:08 AM, Vincent Maillol <vincent...@gmail.com> wrote:
>>
>> Our PEP idea would be to purpose to add a global default value for
>> itemgeet and attrgetter method.
>
> My preference is to not grow that API further. It is creep well beyond its intended uses. At some point, we're really better off just using a lambda.

+1

[Raymond Hettinger]

> On May 2, 2018, at 11:32 PM, Steven D'Aprano <st...@pearwood.info> wrote:
>
> Intended by whom?

By me. I proposed itemgetter() in the first place. That rationale I gave convinced Guido and python-dev to accept it. I then wrote the code, docs, tests and have maintained it for over a decade. So, I have a pretty good idea of what it was intended for.

> I think you are being too dismissive of actual use-cases requested by
> actual users.

Wow, I don't know what to do with this. Over the years, I've added a lot of things requested by users. I really don't like the tone you've struck and what you've implied about me as developer. That feels somewhat pushy and aggressive. Why not just give a +1 to things that are a good idea and -1 for things we're better off without -- no need for ad hominem comments about the person making the post rather than its content -- that feels somewhat disrespectful.

> Default values might not have been the primary use
> considered when the API was first invented, but the fact that people
> keep asking for this feature should tell us that at least some people
> have intended uses that are remaining unmet.

When I've seen the request in the past, it always alway "it might be nice if ..." but there were no legitimate use cases presented, just toy examples. Also, I'm concerned that about increasing the complexity of itemgetter() API to serve an occasional exotic use case rather that being easy to learn and remember for the common cases.


Raymond

[Rob Cliffe via Python-ideas]

>> At some point, we're really better off just using a lambda.
> Maybe I'm slow today, but I'm having trouble seeing how to write this as
> a lambda.

>>> values = {'x': 43, 'y': 55}
>>> x, y, z = (lambda *args : tuple(values.get(arg,0) for arg in
args))('x','y','z')
>>> print(x, y, z)
(43, 55, 0)
>>>

Rob Cliffe

[Guido van Rossum]

Hi Vincent,

Your idea is interesting but we are worried that there are not enough real use cases where it would be useful. Have you encountered situations yourself where this would make a difference? I am asking not for clarifying examples (you already provided one and from that it's perfectly clear to me what you are proposing) but for real-world code that would benefit from this addition to the itemgetter API.

--Guido

--

--Guido van Rossum (python.org/~guido)

[Steven D'Aprano]

On Thu, May 03, 2018 at 04:32:09PM +1000, Steven D'Aprano wrote:

> Maybe I'm slow today, but I'm having trouble seeing how to write this as
> a lambda.

Yes, I was definitely having a "cannot brain, I have the dumb" day,
because it is not that hard to write using lambda. See discussion here:

https://mail.python.org/pipermail/python-list/2018-May/732795.html

If anything, the problem is a plethora of choices, where it isn't clear
which if any is the best way, or the One Obvious Way.

[Raymond Hettinger]

> On May 6, 2018, at 6:00 AM, Steven D'Aprano <st...@pearwood.info> wrote:
>
> On Thu, May 03, 2018 at 04:32:09PM +1000, Steven D'Aprano wrote:
>
>> Maybe I'm slow today, but I'm having trouble seeing how to write this as
>> a lambda.
>
> Yes, I was definitely having a "cannot brain, I have the dumb" day,
> because it is not that hard to write using lambda. See discussion here:
>
> https://mail.python.org/pipermail/python-list/2018-May/732795.html
>
> If anything, the problem is a plethora of choices, where it isn't clear
> which if any is the best way, or the One Obvious Way

At one time, lambda was the one obvious way. Later, partial, itemgetter, attrgetter, and methodcaller were added to express common patterns for key-functions and map(). If needed, the zoo of lambda alternatives could be further extended to add a rpartial() function that partials from the right. That would have helped with Miki's example. Instead of:

get = attrgetter('foo', None)
return get(args) or get(config) or get(env)

He could've written:

get = rpartial(getattr, 'foo', None)
return get(args) or get(config) or get(env)

If itemgetter and attrgetter only did a single lookup, a default might make sense. However, that doesn't fit well with multiple and/or chained lookups where are number of options are possible. (See https://bugs.python.org/issue14384#msg316222 for examples and alternatives).


Raymond

[Vincent Maillol]

Hi everybody,

> Have you encountered situations yourself where this would make a difference ?

I need to get multiple objects in several dicts using the same set of keys.

I wanted to use itemgetter to reduce the number of lines of code but I
have mixed up getattr/dict.get
that have default parameter with attrgetter/itemgetter.

> At some point, we're really better off just using a lambda.

I kept severals line with `.get('key', value)`, I find that is more readable.

> Also, I'm concerned that about increasing the complexity of itemgetter() API to serve an occasional exotic use case rather that being easy to learn and remember for the common cases.

I understand you, each additional parameter increases the cost of
maintenance and update operator module
will take a lot of work we should update c and python module, and
ensure compatibility with pickle.

I did it just to try it with itemgetter

https://github.com/Maillol/cpython/compare/master...Add-default-parameter-to-operator-itemgetter

I don't know if we add parameter `default` to itemgetter, getitem,
attrgetter the API will become
more complexe or more consistency, but I agree with you, it is an
occasional use case and we can always
use `my_dict.get`.

[Danilo J. S. Bellini]

(1)

On 7 May 2018 at 01:07, Raymond Hettinger <raymond....@gmail.com> wrote:

He could've written:

    get = rpartial(getattr, 'foo', None)
    return get(args) or get(config) or get(env)

That's somewhat hybrid, it would behave like:

lambda obj: getattr(obj, "foo", None),

but one might expect a "reversed partial" to have its arguments reversed as well, like:

lambda obj: getattr(obj, None, "foo").

(2)

Why getattr can't accept keyword arguments?

>>> getattr_zero = partial(getattr, default=0)
>>> getattr_zero({}, "blah")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: getattr() takes no keyword arguments

Since partial/partialmethod can apply keyword arguments, allowing the "default=" keyword argument in getattr would be an alternative to the "rpartial".

(3)

At one time, lambda was the one obvious way. [...]

I really like lambdas, but there are some "closure gotchas" like:

>>> for idx in indices:

...     do_something(lambda seq: seq[idx])

If the lambda is stored somewhere to be called after the loop ends (or after its iteration), the seq[idx] would load the item with the last iterated index. This:

>>> for idx in indices:

...     do_something(itemgetter(idx))

would behave more like this instead:

>>> for idx in indices:

...     do_something((lambda i: (lambda seq: seq[i]))(idx))

Which I wouldn't call "the one obvious way".

(4)

If itemgetter and attrgetter only did a single lookup, a default might make sense.  However, that doesn't fit well with multiple and/or chained lookups where are number of options are possible. (See https://bugs.python.org/issue14384#msg316222 for examples and alternatives).

As attrgetter/itemgetter might get heterogeneus data, I would expect a per-get default, not [just] a global default.
Perhaps something like:

>>> itemgetter(-1, 0, -2,
...            default0="first argument default",
...            default1=["second", "argument", "default"],
...            default={"global": "default"},
... )

--

Danilo J. S. Bellini
---------------
"It is not our business to set up prohibitions, but to arrive at conventions." (R. Carnap)

[Vincent Maillol]

As attrgetter/itemgetter might get heterogeneus data, I would expect a per-get default, not [just] a global default.

+1

Perhaps something like:

>>> itemgetter(-1, 0, -2,
...            default0="first argument default",
...            default1=["second", "argument", "default"],
...            default={"global": "default"},
... )

 

The keywords suffixed by indice are uncommon. Maybe we can use dedicated object.

>>> itemgetter(
...     itemgetter.WithDefault(-1, "first argument default"),
...     itemgetter.WithDefault(0, ["second", "argument", "default"])

...     -2  # has no default value

... )

Another possibility is that itemgetter becomes an object with `add_default` method.

>>> getter = itemgetter(-1, 0, 2)
>>> getter.add_default(0, "first argument default")
>>> getter.add_default(2, "third argument default")

or simply default parameter should be a list

>>> itemgetter(-1, 0, 2,

...            default=["first", itemgetter.NoDefault, "third"])

[Steven D'Aprano]

On Tue, May 08, 2018 at 10:13:43AM +0200, Vincent Maillol wrote:

> Perhaps something like:
> >
>
> > >>> itemgetter(-1, 0, -2,
> > ... default0="first argument default",
> > ... default1=["second", "argument", "default"],
> > ... default={"global": "default"},
> > ... )

Sorry Vincent, but I think that's precisely the sort of overly
complicated API that Raymond was worried about when he voted against
this proposal. A single default value that applies when the item (or
key) isn't present is easy to comprehend. But this extention is, I
think, a case of an over-engineered, excessively intricate solution,
and I doubt it solves any real problem.

And I'm not even sure I know what it means. You have keyword parameters
default0, default1, and then *default* with no suffix. Is that a typo
for default2, or is it supposed to be a default default, the default to
use when no default is specified?

> The keywords suffixed by indice are uncommon. Maybe we can use dedicated
> object.
>
> >>> itemgetter(
> ... itemgetter.WithDefault(-1, "first argument default"),
> ... itemgetter.WithDefault(0, ["second", "argument", "default"])
> ... -2 # has no default value
> ... )

I'm afraid there's no elegance to this design either.

I've had many occasions where I want to get an item, falling back on a
default if it is not present. With dicts this is common enough that we
have dict.get(). It is less convenient with lists, and I'd like to see
itemgetter support that case.

But I cannot think of any case where I have needed or wanted to extract

item 5 with "spam" as the default
item 2 with no default
item 3 with "eggs" as the default

(for example), let alone that this is *so common* that I'd rather read
the complicated documenation to work out how to use the function, rather
than just write a short helper:

def extract(sequence):
return ("spam" if len(sequence) < 5 else sequence[5],
sequence[2],
"eggs" if len(sequence) < 5 else sequence[3],
)

Raymond and Serhey are right: beyond a certain point, we should just
write a custom function (whether lambda or not). We just disagree on
which side of that point the single-default value case is, but I think
we will agree that your example is so far past the point that we cannot
even see it from here :-)


--
Steve