[Python-ideas] PEP: Dict addition and subtraction
Steven D'Aprano
discussion.
This should probably go here:
https://github.com/python/peps
but due to technical difficulties at my end, I'm very limited in what I
can do on Github (at least for now). If there's anyone who would like to
co-author and/or help with the process, that will be appreciated.
--
Steven
INADA Naoki
I don't think so. https://bugs.python.org/issue35105 and
https://mail.python.org/pipermail/python-dev/2018-October/155435.html
are about kwargs. I think non string keys are allowed for {**d1,
**d2} by language.
--
INADA Naoki <songof...@gmail.com>
_______________________________________________
Python-ideas mailing list
Python...@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Brandt Bucher
<dict_addition_pep.txt>
Eric V. Smith
I can help you with it. I added it as PEP 584. I had to add the PEP
headers, but didn't do any other editing.
I'm going to be out of town for the next 2 weeks, so I might be slow in
responding.
Eric
Neil Girdhar
I think you should replace all of the lines:
if isinstance(other, dict):
with
if isinstance(self, type(other)):
Since if other is an instance of a dict subclass, he should be the one to process the addition. On the other hand, if self is an instance of the derived type, then we are free to do the combination.
I think you should also change this wording:
"the result type will be the type of the left operand"
since the result type will be negotiated between the operands (even in your implemenation).
__sub__ can be implemented more simply as a dict comprehension.
Don't forget to return self in __isub__ and __iadd__ or they won't work.
I think __isub__ would be simpler like this:
def __isub__(self, it):
if it is self:
self.clear()
else:
for value in it:
del self[value]
return self
I don't see why you would bother looking for keys (iter will do that anyway).
Guido van Rossum
Brett Cannon
I’ve never been part of this process before, but I’m interested in learning and helping any way I can.
My addition implementation is attached to the bpo, and I’m working today on bringing it in line with the PEP in its current form (specifically, subtraction operations).
Brandt Bucher
def __add__(self, other):
if isinstance(other, dict):
new = dict.copy(self)
dict.update(new, other)
return new
return NotImplemented
def __radd__(self, other):
if isinstance(other, dict):
new = dict.copy(other)
dict.update(other, self)
return new
return NotImplemented
def __iadd__(self, other):
if isinstance(other, dict):
dict.update(self, other)
return self
return NotImplemented
This is what my C looks like right now. We can choose to update these semantics to be "nicer" to subclasses, but I don't see any precedent for it (lists, sets, strings, etc.).
Brandt
Neil Girdhar
--
---
You received this message because you are subscribed to a topic in the Google Groups "python-ideas" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/python-ideas/jq5QVTt3CAI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to python-ideas...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
_______________________________________________
Python-ideas mailing list
Python...@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
--
---
You received this message because you are subscribed to a topic in the Google Groups "python-ideas" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/python-ideas/jq5QVTt3CAI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to python-ideas...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Steven D'Aprano
- I'm going to argue for subclass-preserving behaviour;
- I'm not wedded to the idea that dict += should actually call the
update method, so long as it has the same behaviour;
- __iadd__ has no need to return NotImplemented or type-check its
argument.
Details below.
On Fri, Mar 01, 2019 at 04:10:44PM -0800, Brandt Bucher wrote:
[...]
> behave differently from how list subclasses do. List subclasses, without
> overrides, return *list* objects for bare "+" operations
here:
https://mail.python.org/pipermail/python-ideas/2019-March/055547.html
I think the standard handling of subclasses in Python builtins is wrong,
and I don't wish to emulate that wrong behaviour without a really good
reason. Or at least a better reason than "other methods break
subclassing unless explicitly overloaded, so this should do so too".
Or at least not without a fight :-)
> (and "+=" won't call an overridden "extend" method).
docs, I see that they don't actually say that += calls list.extend:
s.extend(t) extends s with the contents of t (for
or s += t the most part the same as s[len(s):len(s)] = t)
https://docs.python.org/3/library/stdtypes.html#mutable-sequence-types
only that they have the same effect. So the wording re lists calling
extend certainly needs to be changed. But that doesn't mean that we must
change the implementation. We have a choice:
- regardless of what lists do, we define += for dicts as literally
calling dict.update; the more I think about it, the less I like this.
- Or we say that += behaves similarly to update, without actually
calling the method. I think I prefer this.
(The second implies either that += either contains a duplicate of the
update logic, or that += and update both delegate to a private, C-level
function that does most of the work.)
I think that the second approach (define += as having the equivalent
semantics of update but without actually calling the update method) is
probably better. That decouples the two methods, allows subclasses to
change one without necessarily changing the other.
> So a more analogous
> pseudo-implementation (if that's what we seek) would look like:
>
> def __add__(self, other):
> if isinstance(other, dict):
> new = dict.copy(self)
> dict.update(new, other)
> return new
> return NotImplemented
The PEP should be more explicit that the approximate implementation does
not imply the copy() and update() methods are actually called.
> def __iadd__(self, other):
> if isinstance(other, dict):
> dict.update(self, other)
> return self
> return NotImplemented
According to PEP 203, which introduced augmented assignment, the
sequence of calls in ``d += e`` is:
1. Try to call ``d.__iadd__(e)``.
2. If __iadd__ is not present, try ``d.__add__(e)``.
3. If __add__ is missing too, try ``e.__radd__(d)``.
but my tests suggest this is inaccurate. I think the correct behaviour
is this:
1. Try to call ``d.__iadd__(e)``.
2. If __iadd__ is not present, or if it returns NotImplemented,
try ``d.__add__(e)``.
3. If __add__ is missing too, or if it returns NotImplemented,
fail with TypeError.
In other words, e.__radd__ is not used.
We don't want dict.__iadd__ to try calling __add__, since the later is
more restrictive and less efficient than the in-place merge. So there is
no need for __iadd__ to return NotImplemented. It should either succeed
on its own, or fail hard:
def __iadd__(self, other):
self.update(other)
return self
Except that the actual C implementation won't call the update method
itself, but will follow the same semantics.
See the docstring for dict.update for details of what is accepted by
update.
--
Steven
Brandt Bucher
James Lu
This way, d1 + d2 isn’t just another obvious way to do {**d1, **d2}. The second syntax makes it clear that a new dictionary is being constructed and that d2 overrides keys from d1.
One can reasonably expect or imagine a situation where a section of code that expects to merge two dictionaries with non-conflicting keys commits a semantic error if it merges two dictionaries with conflicting keys.
To better explain, imagine a program where options is a global variable storing parsed values from the command line.
def verbose_options():
if options.quiet
return {'verbose': True}
def quiet_options():
if options.quiet:
return {'verbose': False}
If we were to define an options() function, return {**quiet_options(), **verbose_options()} implies that verbose overrules quiet; whereas return quiet_options() + verbose_options() implies that verbose and quiet cannot be used simultaneously. I am not aware of another easy way in Python to merge dictionaries while checking for non-conflicting keys.
Compare:
def settings():
return {**quiet_options(), **verbose_options()}
def settings():
try:
return quiet_options() + verbose_options()
except KeyError:
print('conflicting options used', sys.stderr')
sys.exit(1)
***
This is a simple scenario, but you can imagine more complex ones as well. Does —quiet-stage-1 loosen —verbose? Does —quiet-stage-1 conflict with —verbose-stage-1?Does —verbosity=5 override —verbosity=4 or cause an error?
Having {**, **} and + do different things provides a convenient and Pythonic way to model such relationships in code. Indeed, you can even combine the two syntaxes in the same expression to show a mix of overriding and exclusionary behavior.
Anyways, I think it’s a good idea to have this semantic difference in behavior so Python developers have a good way to communicate what is expected of the two dictionaries being merged inside the language. This is like an assertion without
Again, I propose that the + sign merge two python dictionaries such that if there are conflicting keys, a KeyError is thrown, because such “non-conflicting merge” behavior would be useful in Python. It gives clarifying power to the + sign. The + and the {**, **} should serve different roles.
In other words, explicit + is better than implicit {**, **#, unless explicitly suppressed. Here + is explicit whereas {**, **} is implicitly allowing inclusive keys, and the KeyError is expressed suppressed by virtue of not using the {**, **} syntax. People expect the + operator to be commutative, while the {**, **} syntax prompts further examination by virtue of its “weird” syntax.
INADA Naoki
in addition to {**d1, **d2}:
d3 = d1.copy()
d3.update(d2)
It is obvious nor easily discoverable, while it takes two lines.
"There are no obvious way" and "there is at least one obvious way" is
very different.
> Python-ideas mailing list
> Python...@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
INADA Naoki <songof...@gmail.com>
Stefan Behnel
If you need that feature, it can become a new method on dicts.
Stefan
Jimmy Girardet
Hi,
I'm not old on this list but every time there is a proposal, the answer is "what are you trying to solve ?".
Since
z = {**x, **y} and z.update(y)
Exists, I can"t find the answer.
Stefan Behnel
> is "what are you trying to solve ?".
>
> Since
>
I think the main intentions is to close a gap in the language.
[1,2,3] + [4,5,6]
works for lists and tuples,
{1,2,3} | {4,5,6}
works for sets, but joining two dicts isn't simply
{1:2, 3:4} + {5:6}
but requires either some obscure syntax or a statement instead of a simple
expression.
The proposal is to enable the obvious syntax for something that should be
obvious.
Stefan
INADA Naoki
>
> I think the main intentions is to close a gap in the language.
>
> [1,2,3] + [4,5,6]
>
> works for lists and tuples,
>
> {1,2,3} | {4,5,6}
>
> works for sets, but joining two dicts isn't simply
>
> {1:2, 3:4} + {5:6}
>
* Operators are used for concatenate and repeat (Kleene star) in
regular language.
https://en.wikipedia.org/wiki/Regular_language
seq + seq and seq * N are very similar to it, although Python used +
instead of
middle dot (not in ASCII) for concatenate.
* set is directly relating to set in math. | is well known operator for union.
* In case of merging dict, I don't know obvious background in math or
computer science.
So I feel it's very natural that dict don't have operator for merging.
Isn't "for consistency with other types" a wrong consistency?
> but requires either some obscure syntax or a statement instead of a simple
> expression.
>
> The proposal is to enable the obvious syntax for something that should be
> obvious.
Regards,
Jimmy Girardet
> but requires either some obscure syntax or a statement instead of a simple
> expression.
>
> The proposal is to enable the obvious syntax for something that should be
> obvious.
>
> Stefan
The discussions on this list show that the behavior of `+` operator with
dict will never be obvious (first wins or second wins or add results or
raise Exception). So the user will always have to look at the doc or
test it to know the intended behavior.
That said, [1,2] + [3] equals [1,2,3] but not[1,2, [3]] and that was
not obvious to me, and I survived.
James Lu
> On Mar 4, 2019, at 3:41 AM, Stefan Behnel <stef...@behnel.de> wrote:
>
> James Lu schrieb am 04.03.19 um 03:28:
>> I propose that the + sign merge two python dictionaries such that if there are conflicting keys, a KeyError is thrown.
>
> Please, no. That would be really annoying.
>
> If you need that feature, it can become a new method on dicts.
>
> Stefan
Stefan Behnel
enough" because it's a statement and not an expression. It does not replace
the convenience of an expression.
Stefan
James Lu
On Mar 4, 2019, at 4:51 AM, Stefan Behnel <stef...@behnel.de> wrote:
Jimmy Girardet schrieb am 04.03.19 um 10:12:I'm not old on this list but every time there is a proposal, the answeris "what are you trying to solve ?".Since|z ={**x,**y} and z.update(y) Exists, I can"t find the answer.
I think the main intentions is to close a gap in the language.
[1,2,3] + [4,5,6]
works for lists and tuples,
{1,2,3} | {4,5,6}
works for sets, but joining two dicts isn't simply
{1:2, 3:4} + {5:6}
but requires either some obscure syntax or a statement instead of a simple
expression.
The proposal is to enable the obvious syntax for something that should be
obvious.
James Lu
> On Mar 4, 2019, at 10:02 AM, Stefan Behnel <stef...@behnel.de> wrote:
>
> INADA Naoki schrieb am 04.03.19 um 11:15:
>> Why statement is not enough?
>
> I'm not sure I understand why you're asking this, but a statement is "not
> enough" because it's a statement and not an expression. It does not replace
> the convenience of an expression.
>
> Stefan
Steven D'Aprano
> If you want to merge it without a KeyError, learn and use the more explicit {**d1, **d2} syntax.
In other words, explicit + is better than implicit {**, **#, unless
explicitly suppressed. Here + is explicit whereas {**, **} is
implicitly allowing inclusive keys, and the KeyError is expressed
suppressed by virtue of not using the {**, **} syntax.
you cannot even decided which is which.
--
Steven
Steven D'Aprano
> How many situations would you need to make a copy of a dictionary and
> then update that copy and override old keys from a new dictionary?
Very frequently.
That's why we have a dict.update method, which if I remember correctly,
was introduced in Python 1.5 because people were frequently re-inventing
the same wheel:
def update(d1, d2):
for key in d2.keys():
d1[key] in d2[key]
You should have a look at how many times it is used in the standard
library:
[steve@ando cpython]$ cd Lib/
[steve@ando Lib]$ grep -U "\.update[(]" *.py */*.py | wc -l
373
Now some of those are false positives (docstrings, comments, non-dicts,
etc) but that still leaves a lot of examples of wanting to override old
keys. This is a very common need. Wanting an exception if the key
already exists is, as far as I can tell, very rare.
It is true that many of the examples in the std lib involve updating an
existing dict, not creating a new one. But that's only to be expected:
since Python didn't provide an obvious functional version of update,
only an in-place version, naturally people get used to writing
in-place code.
(Think about how long we made do without sorted(). I don't know about
other people, but I now find sorted indispensible, and probably use it
ten or twenty times more often than the in-place version.)
[...]
> The KeyError of my proposal is a feature, a sign that something is
> wrong, a sign an invariant is being violated.
Why is "keys are unique" an invariant?
The PEP gives a good example of when this "invariant" would be
unnecessarily restrictive:
For example, updating default configuration values with
user-supplied values would most often fail under the
requirement that keys are unique::
prefs = site_defaults + user_defaults + document_prefs
Another example would be when reading command line options, where the
most common convention is for "last option seen" to win:
[steve@ando Lib]$ grep --color=always --color=never "zero" f*.py
fileinput.py: numbers are zero; nextfile() has no effect.
fractions.py: # the same way for any finite a, so treat a as zero.
functools.py: # prevent their ref counts from going to zero during
and the output is printed without colour.
(I've slightly edited the above output so it will fit in the email
without wrapping.)
The very name "update" should tell us that the most useful behaviour is
the one the devs decided on back in 1.5: have the last seen value win.
How can you update values if the operation raises an error if the key
already exists? If this behaviour is ever useful, I would expect that it
will be very rare.
An update or merge is effectively just running through a loop setting
the value of a key. See the pre-Python 1.5 function above. Having update
raise an exception if the key already exists would be about as useful as
having ``d[key] = value`` raise an exception if the key already exists.
Unless someone can demonstrate that the design of dict.update() was a
mistake, and the "require unique keys" behaviour is more common, then
I maintain that for the very rare cases you want an exception, you can
subclass dict and overload the __add__ method:
# Intentionally simplified version.
def __add__(self, other):
if self.keys() & other.keys():
raise KeyError
return super().__add__(self, other)
> The ugliness of the syntax makes one pause
> and think and ask: “Why is it important that the keys from this
> dictionary override the ones from another dictionary?”
Because that is the most common and useful behaviour. That's what it
means to *update* a dict or database, and this proposal is for an update
operator.
The ugliness of the existing syntax is not a feature, it is a barrier.
--
Steven
Rhodri James
>
>> On Mar 4, 2019, at 10:02 AM, Stefan Behnel <stef...@behnel.de> wrote:
>>
>> INADA Naoki schrieb am 04.03.19 um 11:15:
>>> Why statement is not enough?
>>
>> I'm not sure I understand why you're asking this, but a statement is "not
>> enough" because it's a statement and not an expression. It does not replace
>> the convenience of an expression.
>>
>> Stefan
> There is already an expression for key-overriding merge. Why do we need a new one?
--
Rhodri James *-* Kynesim Ltd
Del Gan
> the augmented assignment version allows anything the ``update`` method allows, such as iterables of key/value pairs
I am a little surprised by this choice.
First, this means that "a += b" would not be equivalent to "a = a +
b". Is there other built-in types which act differently if called with
the operator or augmented assignment version?
Secondly, that would imply I would no longer be able to infer the type
of "a" while reading "a += [('foo', 'bar')]". Is it a list? A dict?
Those two points make me uncomfortable with "+=" strictly behaving
like ".update()".
2019-03-04 17:44 UTC+01:00, Rhodri James <rho...@kynesim.co.uk>:
Guido van Rossum
> the augmented assignment version allows anything the ``update`` method allows, such as iterables of key/value pairs
I am a little surprised by this choice.
First, this means that "a += b" would not be equivalent to "a = a +
b". Is there other built-in types which act differently if called with
the operator or augmented assignment version?
>>> a = [1]
>>> a + 'a'
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: can only concatenate list (not "str") to list
>>> a += 'a'
>>> a
[1, 'a']
>>>
Secondly, that would imply I would no longer be able to infer the type
of "a" while reading "a += [('foo', 'bar')]". Is it a list? A dict?
Those two points make me uncomfortable with "+=" strictly behaving
like ".update()".
James Lu
On Mon, Mar 04, 2019 at 10:01:23AM -0500, James Lu wrote:
If you want to merge it without a KeyError, learn and use the more explicit {**d1, **d2} syntax.
In your previous email, you said the {**d ...} syntax was implicit:
In other words, explicit + is better than implicit {**, **#, unless
explicitly suppressed. Here + is explicit whereas {**, **} is
implicitly allowing inclusive keys, and the KeyError is expressed
suppressed by virtue of not using the {**, **} syntax.
It is difficult to take your "explicit/implicit" argument seriously when
you cannot even decided which is which.
In your previous email, you said the {**d ...} syntax was implicit:
In other words, explicit + is better than implicit {**, **#, unless
explicitly suppressed. Here + is explicit whereas {**, **} is
implicitly allowing inclusive keys, and the KeyError is expressed
suppressed by virtue of not using the {**, **} syntax.
It is difficult to take your "explicit/implicit" argument seriously when
you cannot even decided which is which.
James Lu
How many situations would you need to make a copy of a dictionary andthen update that copy and override old keys from a new dictionary?
Very frequently.
That's why we have a dict.update method, which if I remember correctly,
was introduced in Python 1.5 because people were frequently re-inventing
the same wheel:
def update(d1, d2):
for key in d2.keys():
d1[key] in d2[key]
You should have a look at how many times it is used in the standard
library:
[steve@ando cpython]$ cd Lib/
[steve@ando Lib]$ grep -U "\.update[(]" *.py */*.py | wc -l
373
Now some of those are false positives (docstrings, comments, non-dicts,
etc) but that still leaves a lot of examples of wanting to override old
keys. This is a very common need. Wanting an exception if the key
already exists is, as far as I can tell, very rare.
It is true that many of the examples in the std lib involve updating an
existing dict, not creating a new one. But that's only to be expected:
since Python didn't provide an obvious functional version of update,
only an in-place version, naturally people get used to writing
in-place code.
James Lu
> On Mar 4, 2019, at 11:25 AM, Steven D'Aprano <st...@pearwood.info> wrote:
>
> The PEP gives a good example of when this "invariant" would be
> unnecessarily restrictive:
>
> For example, updating default configuration values with
> user-supplied values would most often fail under the
> requirement that keys are unique::
>
> prefs = site_defaults + user_defaults + document_prefs
>
>
> Another example would be when reading command line options, where the
> most common convention is for "last option seen" to win:
>
> [steve@ando Lib]$ grep --color=always --color=never "zero" f*.py
> fileinput.py: numbers are zero; nextfile() has no effect.
> fractions.py: # the same way for any finite a, so treat a as zero.
> functools.py: # prevent their ref counts from going to zero during
>
> and the output is printed without colour.
>
> (I've slightly edited the above output so it will fit in the email
> without wrapping.)
>
> The very name "update" should tell us that the most useful behaviour is
> the one the devs decided on back in 1.5: have the last seen value win.
> How can you update values if the operation raises an error if the key
> already exists? If this behaviour is ever useful, I would expect that it
> will be very rare.
> An update or merge is effectively just running through a loop setting
> the value of a key. See the pre-Python 1.5 function above. Having update
> raise an exception if the key already exists would be about as useful as
> having ``d[key] = value`` raise an exception if the key already exists.
>
> Unless someone can demonstrate that the design of dict.update() was a
> mistake
You’re making a logical mistake here. + isn’t supposed to have .update’s behavior and it never was supposed to.
> , and the "require unique keys" behaviour is more common,
I just have. 99% of the time you want to have keys from one dict override another, you’d be better off doing it in-place and so would be using .update() anyways.
> then
> I maintain that for the very rare cases you want an exception, you can
> subclass dict and overload the __add__ method:
Well, yes, the whole point is to define the best default behavior.
James Lu
David Foster
have seen almost no mention of dict difference.
This lack of discussion interest combined with me not recalling having
needed the proposed subtraction semantics personally makes me wonder if
we should hold off on locking in subtraction semantics just yet. Perhaps
we could just scope the proposal to dictionary addition only for now?
If I *were* to define dict difference, my intuition suggests supporting
a second operand that is any iterable of keys and not just dicts.
(Augmented dict subtraction is already proposed to accept such a broader
second argument.)
David Foster | Seattle, WA, USA
On 3/1/19 8:26 AM, Steven D'Aprano wrote:
> Attached is a draft PEP on adding + and - operators to dict for
> discussion.
>
> This should probably go here:
>
> https://github.com/python/peps
>
> but due to technical difficulties at my end, I'm very limited in what I
> can do on Github (at least for now). If there's anyone who would like to
> co-author and/or help with the process, that will be appreciated.
>
>
>
>
Brandt Bucher
It’s odd that RHS values are not used at all, period. Further, there’s no precedent for bulk sequence/mapping removals like this... except for sets, for which it is certainly justified.
I’ve had the opportunity to play around with my reference implementation over the last few days, and despite my initial doubts, I have *absolutely* fallen in love with dictionary addition — I even accidentally tried to += two dictionaries at work on Friday (a good, but frustrating, sign). For context, I was updating a module-level mapping with an imported one, a use case I hadn’t even previously considered.
I have tried to fall in love with dict subtraction the same way, but every code sketch/test I come up with feels contrived and hack-y. I’m indifferent towards it, at best.
TL;DR: I’ve lived with both for a week. Addition is now habit, subtraction is still weird.
> Nice branch name! :)
I couldn’t help myself.
Brandt
INADA Naoki
>
> INADA Naoki schrieb am 04.03.19 um 11:15:
> > Why statement is not enough?
>
> I'm not sure I understand why you're asking this, but a statement is "not
> enough" because it's a statement and not an expression. It does not replace
> the convenience of an expression.
>
> Stefan
>
What is "convenience of an expression"?
Is it needed to make Python more readable language?
Anyway, If "there is expression" is the main reason for this proposal, symbolic
operator is not necessary.
`new = d1.updated(d2)` or `new = dict.merge(d1, d2)` are enough.
Python preferred name over symbol in general.
Symbols are readable and understandable only when it has good
math metaphor.
Sets has symbol operator because it is well known in set in math,
not because set is frequently used.
In case of dict, there is no simple metaphor in math.
It just cryptic and hard to Google.
--
INADA Naoki <songof...@gmail.com>
Jonathan Fine
I'm grateful to Steve for preparing the current draft. Thank you.
It's strong on implementation, but I find it weak on motivation. I
hope that when time is available you (and the other contributors)
could transfer some motivating material into the PEP, from
python-ideas.
According to PEP 001, the PEP "should clearly explain why the existing
language specification is inadequate to address the problem that the
PEP solves". So it is important.
--
Jonathan
Steven D'Aprano
> > If the keys are not strings, it currently works in CPython, but it may not work with other implementations, or future versions of CPython[2].
>
> I don't think so. https://bugs.python.org/issue35105 and
> https://mail.python.org/pipermail/python-dev/2018-October/155435.html
> are about kwargs. I think non string keys are allowed for {**d1,
> **d2} by language.
Is this documented somewhere?
Or is there a pronouncement somewhere that it is definitely expected to
work in any language calling itself Python?
Thanks,
--
Steven
Inada Naoki
>
> On Sat, Mar 02, 2019 at 01:47:37AM +0900, INADA Naoki wrote:
> > > If the keys are not strings, it currently works in CPython, but it may not work with other implementations, or future versions of CPython[2].
> >
> > I don't think so. https://bugs.python.org/issue35105 and
> > https://mail.python.org/pipermail/python-dev/2018-October/155435.html
> > are about kwargs. I think non string keys are allowed for {**d1,
> > **d2} by language.
>
> Is this documented somewhere?
>
dict display supported non-string keys from very old.
I believe {3: 4} is supported by Python language, not CPython
implementation behavior.
https://docs.python.org/3/reference/expressions.html#grammar-token-dict-display
> Or is there a pronouncement somewhere that it is definitely expected to
> work in any language calling itself Python?
>
>
> Thanks,
>
>
>
> --
> Steven
> _______________________________________________
> Python-ideas mailing list
> Python...@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
Inada Naoki <songof...@gmail.com>
David Shawley
I think the main intentions is to close a gap in the language.
[1,2,3] + [4,5,6]
works for lists and tuples,
{1,2,3} | {4,5,6}
works for sets, but joining two dicts isn't simply
{1:2, 3:4} + {5:6}
but requires either some obscure syntax or a statement instead of a simple
expression.
The proposal is to enable the obvious syntax for something that should be
obvious.
failed, as PEP 8 itself states that the rules can be broken as needed. - Paul Moore.
Jimmy Girardet
Does anyone have an example of another programming language thatallows for addition of dictionaries/mappings?
fun main() {
var a = mutableMapOf<String,Int>("a" to 1, "b" to 2)
var b = mutableMapOf<String,Int>("c" to 1, "b" to 3)
println(a)
println(b)
println(a + b)
println(b + a)
}
{a=1, b=2}
{c=1, b=3}
{a=1, b=3, c=1}
{c=1, b=2, a=1}
Steven D'Aprano
> "Putting Metaclasses to Work" (ISBN-13 978-0201433050) presents a more
> mathematical view of programming language types that includes two
> distinct operations for combining dictionaries -- merge and recursive
> merge.
>
> For two input dictionaries D1 & D2 and the output dictionary O
>
> D1 merge D2
> O is D1 with the of those keys of D2 that do not have keys in D1
>
> D1 recursive-merge D2
> For all keys k, O[k] = D1[k] recursive merge D2[k] if both D1[k]
> and D2[k] are dictionaries, otherwise O[k] = (D1 merge D2)[k].
suspect that you've left at least one word out of the first.
Fortunately your example below is extremely clear, thank you.
[...]
PEP proposes as that seems to be the most frequently requested
behaviour. It is also the only behaviour which has been deemed useful
enough in nearly 30 years of Python's history to be added to dict as a
method.
> >>> merge(d1, d2)
> {'publisher': 'Academic Press',
> 'title': 'Structured Programming',
> 'locations': {'Dijkstra': 'University of Texas',
> 'Hoare': 'Oxford University',
> 'Dahl': 'University of Oslo'},
> 'authors': 'Dahl, Dijkstra, and Hoare'}
using ChainMap or the proposed dict difference operator:
dict( ChainMap(d1, d2) )
# or
d1 + (d2 - d1)
or simply by swapping the order of the operands:
d2 + d1
(These are not *identical* in effect, there are small differences with
respect to key:value identity, and order of keys. But they ought to give
*equal* results.)
Personally, I don't think that behaviour is as useful as the first, but
it is certainly a legitimate kind of merge.
As far as I know, this has never been requested before. Perhaps it is
too niche?
> >>> recursive_merge(d1, d2)
> {'publisher': 'Academic Press',
> 'title': 'Structured Programming',
> 'locations': {'North America': 'New York',
> 'Europe': 'London',
> 'Dijkstra': 'University of Texas',
> 'Hoare': 'Oxford University',
> 'Dahl': 'University of Oslo'},
> 'authors': 'Dahl, Dijkstra, and Hoare'}
def merge(a, b):
new = a.copy()
for key, value in b:
if key not in a:
# Add new keys.
new[key] = value
else:
v = new[key]
if isinstance(value, dict) and isinstance(v, dict):
# If both values are dicts, merge them.
new[key] = merge(v, value)
else:
# What to do if only one is a dict?
# Or if neither is a dict?
return new
I've seen variants of this where duplicate keys are handled by building
a list of the values:
def merge(a, b):
new = a.copy()
for key, value in b:
if key in a:
v = new[key]
if isinstance(v, list):
v.append(value)
else:
new[key] = [v, value]
...
or by concatenating values, or adding them (as Counter does), etc. We
have subclasses and operator overloading, so you can implement whatever
behaviour you like.
The question is, is this behaviour useful enough and common enough to be
built into dict itself?
> IMO, having more than one obvious outcome means that we should refuse
> the temptation to guess.
Nobody says:
When I print some strings, I can seperate them with spaces, or
dots, or newlines, and print a newline at the end, or suppress
the newline. Since all of these behaviours might be useful for
somebody, we should not "guess" what the user wants. Therefore
we should not have a print() function at all.
The behaviour of print() is not a guess as to what the user wants. We
offer a specific behaviour, and if the user is happy with that, then
they can use print(), and if not, they can write their own.
The same applies here: we're offering one specific behaviour that we
think is the most important, and anyone who wants another can write
their own.
If people don't like my choice of what I think is the most important
(copy-and-update, with last seen wins), they can argue for whichever
alternative they like. If they make a convincing enough case, the PEP
can change :-)
James Lu has already tried to argue that the "raise on non-unique keys"
is the best behaviour. I have disagreed with that, but if James makes a
strong enough case for his idea, and it gains sufficient support, I
could be persuaded to change my position.
Or he can write a competing PEP and the Steering Council can decide
between the two ideas.
> If we do, then the result is only obvious
> to a subset of users and will be a surprise to the others.
assumptions about behaviour based on their own wild guesses.
We should get away from the idea that the only behaviours we can provide
are those which are "obvious" (intuitive?) to people who guess what it
means without reading the docs. It's great when a function's meaning can
be guessed or inferred from a basic understanding of English:
len(string) # assuming len is an abbreviation for length
but that sets the bar impossibly high. We can't guess what these do, not
with any precision:
print(spam, eggs) # prints spaces between arguments or not?
spam is eggs # that's another way of spelling == right?
zip(spam, eggs) # what does it do if args aren't the same length?
and who can guess what these do without reading the docs?
property, classmethod, slice, enumerate, iter
I don't think that Python is a worse language for having specified a
meaning for these rather than leaving them out.
The Zen's prohibition against guessing in the face of ambiguity does not
mean that we must not add a feature to the language that requires the
user to learn what it does first.
> It's also useful to note that I am having trouble coming up with
> another programming language that supports a "+" operator for map types.
>
> Does anyone have an example of another programming language that
> allows for addition of dictionaries/mappings?
>
> If so, what is the behavior there?
here so I'm going to take this opportunity to go to bed :-)
--
Steven
Guido van Rossum
On Tue, Mar 5, 2019 at 7:26 PM Steven D'Aprano <st...@pearwood.info> wrote:
>
> On Sat, Mar 02, 2019 at 01:47:37AM +0900, INADA Naoki wrote:
> > > If the keys are not strings, it currently works in CPython, but it may not work with other implementations, or future versions of CPython[2].
> >
> > I don't think so. https://bugs.python.org/issue35105 and
> > https://mail.python.org/pipermail/python-dev/2018-October/155435.html
> > are about kwargs. I think non string keys are allowed for {**d1,
> > **d2} by language.
>
> Is this documented somewhere?
It is not explicitly documented. But unlike keyword argument,
dict display supported non-string keys from very old.
I believe {3: 4} is supported by Python language, not CPython
implementation behavior.
https://docs.python.org/3/reference/expressions.html#grammar-token-dict-display
Anders Hovmöller
> I'd like to remove all doubt: {**d1} needs to work regardless of the key type, as long as it's hashable (d1 could be some mapping implemented without hashing, e.g. using a balanced tree, so that it could support unhashable keys).
>
> If there's doubt about this anywhere, we could add an example to the docs and to the PEP.
/ Anders
Guido van Rossum
On a related note: **kwargs, should they support arbitrary strings as keys? I depend on this behavior in production code and all python implementations handle it.
Inada Naoki
Del Gan
>> the operator or augmented assignment version?
>
2019-03-05 0:57 UTC+01:00, Guido van Rossum <gu...@python.org>:
> works:
Python since years.
Thanks for the clarification. This makes perfect sense for += to
behaves like .update() then.
Greg Ewing
> The question is, is [recursive merge] behaviour useful enough and
values out of many. I think it would be better to provide
a merge function or method that lets you specify a function
for merging values.
--
Greg
Brandt Bucher
These semantics are intended to match those of update as closely as possible. For the dict built-in itself, calling keys is redundant as iteration over a dict iterates over its keys; but for subclasses or other mappings, update prefers to use the keys method.The above paragraph may be inaccurate. Although the dict docstring states that keys will be called if it exists, this does not seem to be the case for dict subclasses. Bug or feature?
D.update([E, ]**F) -> None. Update D from dict/iterable E and F.
If E is present and has a .keys() method, then does: for k in E: D[k] = E[k]
If E is present and lacks a .keys() method, then does: for k, v in E: D[k] = v
In either case, this is followed by: for k in F: D[k] = F[k]
If E is present, has a .keys() method, and is a subclass of dict, then does: for k in E: D[k] = E[k]
If E is present, has a .keys() method, and is not a subclass of dict, then does: for k in E.keys(): D[k] = E[k]
Attached is a draft PEP on adding + and - operators to dict for
discussion.
This should probably go here:
https://github.com/python/peps
but due to technical difficulties at my end, I'm very limited in what I
can do on Github (at least for now). If there's anyone who would like to
co-author and/or help with the process, that will be appreciated.
--
Steven
Brandt Bucher
Should our __sub__ behavior be the same...
Brandt Bucher
If E is present, has a .keys() method, and is not a subclass of dict or has overridden __iter__, then does: for k in E.keys(): D[k] = E[k]
Steven D'Aprano
> I propose that the + sign merge two python dictionaries such that if
> there are conflicting keys, a KeyError is thrown.
This proposal is for a simple, operator-based equivalent to
dict.update() which returns a new dict. dict.update has existed since
Python 1.5 (something like a quarter of a century!) and never grown a
"unique keys" version.
I don't recall even seeing a request for such a feature. If such a
unique keys version is useful, I don't expect it will be useful often.
> This way, d1 + d2 isn’t just another obvious way to do {**d1, **d2}.
One of the reasons for preferring + is that it is an obvious way to do
something very common, while {**d1, **d2} is as far from obvious as you
can get without becoming APL or Perl :-)
If I needed such a unique key version of update, I'd use a subclass:
class StrictDict(dict):
def __add__(self, other):
if isinstance(other, dict) and (self.keys() & other.keys()):
raise KeyError('non-unique keys')
return super().__add__(self, other)
# and similar for __radd__.
rather than burden the entire language, and every user of it, with
having to learn the subtle difference between the obvious + operator and
the error-prone and unobvious trick of {*d1, *d2}.
( Did you see what I did there? *wink* )
> The second syntax makes it clear that a new dictionary is being
> constructed and that d2 overrides keys from d1.
Only because you have learned the rule that {**d, **e) means to
construct a new dict by merging, with the rule that in the event of
duplicate keys, the last key seen wins. If you hadn't learned that rule,
there is nothing in the syntax which would tell you the behaviour. We
could have chosen any rule we liked:
- raise an exception, like you get a TypeError if you pass the
same keyword argument to a function twice: spam(foo=1, foo=2);
- first value seen wins;
- last value seen wins;
- random value wins;
- anything else we liked!
There is nothing "clear" about the syntax which makes it obvious which
behaviour is implemented. We have to learn it.
> One can reasonably expect or imagine a situation where a section of
> code that expects to merge two dictionaries with non-conflicting keys
> commits a semantic error if it merges two dictionaries with
> conflicting keys.
I can imagine it, but I don't think I've ever needed it, and I can't
imagine wanting it often enough to wish it was not just a built-in
function or method, but actual syntax.
Do you have some real examples of wanting an error when trying to update
a dict if keys match?
> To better explain, imagine a program where options is a global
> variable storing parsed values from the command line.
>
> def verbose_options():
> if options.quiet
> return {'verbose': True}
>
> def quiet_options():
> if options.quiet:
> return {'verbose': False}
That seems very artifical to me. Why not use a single function:
def verbose_options(): # There's more than one?
return {'verbose': not options.quiet}
The way you have written those functions seems weird to me. You already
have a nice options object, with named fields like "options.quiet", why
are you turning it into not one but *two* different dicts, both
reporting the same field?
And its buggy: if options.quiet is True, then the key 'quiet'
should be True, not the 'verbose' key.
Do you have *two* functions for every preference setting that takes a
true/false flag?
What do you do for preference settings that take multiple values? Create
a vast number of specialised functions, one for each possible value?
def A4_page_options():
if options.page_size == 'A4':
return {'page_size': 'A4'}
def US_Letter_page_options():
if options.page_size == 'US Letter':
return {'page_size': 'US Letter'}
page_size = (
A4_page_options() + A3_page_options() + A5_page_options()
+ Foolscape_page_options + Tabloid_page_options()
+ US_Letter_page_options() + US_Legal_page_options()
# and about a dozen more...
)
The point is, although I might be wrong, I don't think that this example
is a practical, realistic use-case for a unique keys version of update.
To me, your approach seems so complicated and artificial that it seems
like it was invented specifically to justify this "unique key" operator,
not something that we would want to write in real life.
But even if it real code, the question is not whether it is EVER useful
for a dict update to raise an exception on matching keys. The question
is whether this is so often useful that this is the behaviour we want to
make the default for dicts.
[...]
> Again, I propose that the + sign merge two python dictionaries such
> that if there are conflicting keys, a KeyError is thrown, because such
> “non-conflicting merge” behavior would be useful in Python.
I don't think it would be, at least not often.
If it were common enough to justify a built-in operator to do this, we
would have had many requests for a dict.unique_update or similar by now,
and I don't think we have.
> It gives
> clarifying power to the + sign. The + and the {**, **} should serve
> different roles.
>
> In other words, explicit + is better than implicit {**, **#, unless
> explicitly suppressed. Here + is explicit whereas {**, **} is
> implicitly allowing inclusive keys,
If I had a cent for every time people misused "explicit" to mean "the
proposal that I like", I'd be rich.
In what way is the "+" operator *explicit* about raising an exception on
duplicate keys? These are both explicit:
merge_but_raise_exception_if_any_duplicates(d1, d2)
merge(d1, d2, raise_if_duplicates=True)
and these are both equally implicit:
d1 + d2
{**d1, **d2}
since the behaviour on duplicates is not explicitly stated in clear and
obvious language, but implied by the rules of the language.
[...]
> People expect the + operator to be commutative
THey are wrong to expect that, because the + operator is already not
commutative for:
str
bytes
bytearray
list
tuple
array.array
collections.deque
collections.Counter
and possibly others.
Steven D'Aprano
>
> > On Mar 4, 2019, at 11:25 AM, Steven D'Aprano <st...@pearwood.info> wrote:
> > Another example would be when reading command line options, where the
> > most common convention is for "last option seen" to win:
> >
> > [steve@ando Lib]$ grep --color=always --color=never "zero" f*.py
[...]
> Indeed, in this case you would want to use {**, **} syntax.
No I would NOT want to use the {**, **} syntax, because it is ugly.
That's why people ask for + instead. (Or perhaps I should say "as well
as" since the double-star syntax is not going away.)
[...]
> > Unless someone can demonstrate that the design of dict.update() was a
> > mistake
>
> You’re making a logical mistake here. + isn’t supposed to have
> .update’s behavior and it never was supposed to.
James, I'm the author of the PEP, and for the purposes of the proposal,
the + operator is supposed to do what I say it is supposed to do.
You might be able to persuade me to change the PEP, if you have a
sufficiently good argument, or you can write your own counter PEP making
a different choice, but please don't tell me what I intended. I know
what I intended, and it is for + to have the same last-key-wins
behaviour as update. That's the behaviour which is most commonly
requested in the various times this comes up.
> > , and the "require unique keys" behaviour is more common,
>
> I just have.
No you haven't -- you have simply *declared* that it is more common,
without giving any evidence for it.
> 99% of the time you want to have keys from one dict override another,
> you’d be better off doing it in-place and so would be using .update()
> anyways.
I don't know if it is "99% of the time" or 50% of the time or 5%,
but this PEP is for the remaining times where we don't want in-place
updates but we want a new dict.
I use list.append or list.extend more often than list concatenation, but
when I want a new list, list concatenation is very useful. This proposal
is about those cases where we want a new dict.
--
Steven
Josh Rosenberg
On Sun, Mar 03, 2019 at 09:28:30PM -0500, James Lu wrote:
> I propose that the + sign merge two python dictionaries such that if
> there are conflicting keys, a KeyError is thrown.
This proposal is for a simple, operator-based equivalent to
dict.update() which returns a new dict. dict.update has existed since
Python 1.5 (something like a quarter of a century!) and never grown a
"unique keys" version.
I don't recall even seeing a request for such a feature. If such a
unique keys version is useful, I don't expect it will be useful often.
> This way, d1 + d2 isn’t just another obvious way to do {**d1, **d2}.
One of the reasons for preferring + is that it is an obvious way to do
something very common, while {**d1, **d2} is as far from obvious as you
can get without becoming APL or Perl :-)
"The Zen's prohibition against guessing in the face of ambiguity does not
mean that we must not add a feature to the language that requires the
user to learn what it does first."
> The second syntax makes it clear that a new dictionary is being
> constructed and that d2 overrides keys from d1.
Only because you have learned the rule that {**d, **e) means to
construct a new dict by merging, with the rule that in the event of
duplicate keys, the last key seen wins. If you hadn't learned that rule,
there is nothing in the syntax which would tell you the behaviour. We
could have chosen any rule we liked:
Guido van Rossum
On Tue, Mar 5, 2019 at 11:16 PM Steven D'Aprano <st...@pearwood.info> wrote:On Sun, Mar 03, 2019 at 09:28:30PM -0500, James Lu wrote:
> I propose that the + sign merge two python dictionaries such that if
> there are conflicting keys, a KeyError is thrown.
This proposal is for a simple, operator-based equivalent to
dict.update() which returns a new dict. dict.update has existed since
Python 1.5 (something like a quarter of a century!) and never grown a
"unique keys" version.
I don't recall even seeing a request for such a feature. If such a
unique keys version is useful, I don't expect it will be useful often.I have one argument in favor of such a feature: It preserves concatenation semantics. + means one of two things in all code I've ever seen (Python or otherwise):1. Numeric addition (including element-wise numeric addition as in Counter and numpy arrays)2. Concatenation (where the result preserves all elements, in order, including, among other guarantees, that len(seq1) + len(seq2) == len(seq1 + seq2))dict addition that didn't reject non-unique keys wouldn't fit *either* pattern; the main proposal (making it equivalent to left.copy(), followed by .update(right)) would have the left hand side would win on ordering, the right hand side on values, and wouldn't preserve the length invariant of concatenation. At least when repeated keys are rejected, most concatenation invariants are preserved; order is all of the left elements followed by all of the right, and no elements are lost.
Josh Rosenberg
On Tue, Mar 5, 2019 at 3:50 PM Josh Rosenberg <shadowranger...@gmail.com> wrote:On Tue, Mar 5, 2019 at 11:16 PM Steven D'Aprano <st...@pearwood.info> wrote:On Sun, Mar 03, 2019 at 09:28:30PM -0500, James Lu wrote:
> I propose that the + sign merge two python dictionaries such that if
> there are conflicting keys, a KeyError is thrown.
This proposal is for a simple, operator-based equivalent to
dict.update() which returns a new dict. dict.update has existed since
Python 1.5 (something like a quarter of a century!) and never grown a
"unique keys" version.
I don't recall even seeing a request for such a feature. If such a
unique keys version is useful, I don't expect it will be useful often.I have one argument in favor of such a feature: It preserves concatenation semantics. + means one of two things in all code I've ever seen (Python or otherwise):1. Numeric addition (including element-wise numeric addition as in Counter and numpy arrays)2. Concatenation (where the result preserves all elements, in order, including, among other guarantees, that len(seq1) + len(seq2) == len(seq1 + seq2))dict addition that didn't reject non-unique keys wouldn't fit *either* pattern; the main proposal (making it equivalent to left.copy(), followed by .update(right)) would have the left hand side would win on ordering, the right hand side on values, and wouldn't preserve the length invariant of concatenation. At least when repeated keys are rejected, most concatenation invariants are preserved; order is all of the left elements followed by all of the right, and no elements are lost.I must by now have seen dozens of post complaining about this aspect of the proposal. I think this is just making up rules (e.g. "+ never loses information") to deal with an aspect of the design where a *choice* must be made. This may reflect the Zen of Python's "In the face of ambiguity, refuse the temptation to guess." But really, that's a pretty silly rule (truly, they aren't all winners). Good interface design constantly makes choices in ambiguous situations, because the alternative is constantly asking, and that's just annoying.We have a plethora of examples (in fact, almost all alternatives considered) of situations related to dict merging where a choice is made between conflicting values for a key, and it's always the value further to the right that wins: from d[k] = v (which overrides the value when k is already in the dict) to d1.update(d2) (which lets the values in d2 win), including the much lauded {**d1, **d2} and even plain {'a': 1, 'a': 2} has a well-defined meaning where the latter value wins.
As to why raising is worse: First, none of the other situations I listed above raises for conflicts. Second, there's the experience of str+unicode in Python 2, which raises if the str argument contains any non-ASCII bytes. In fact, we disliked it so much that we changed the language incompatibly to deal with it.
Stefan Behnel
>> INADA Naoki schrieb am 04.03.19 um 11:15:
>>> Why statement is not enough?
>>
>> I'm not sure I understand why you're asking this, but a statement is
>> "not enough" because it's a statement and not an expression. It does
>> not replace the convenience of an expression.
>
> What is "convenience of an expression"?
the thing you need, rather than having to split code into statements that
create it step by step before you can use it.
Think of comprehensions versus for-loops. Comprehensions are expressions
that don't add anything to the language that a for-loop cannot achieve.
Still, everyone uses them because they are extremely convenient.
> Is it needed to make Python more readable language?
> Anyway, If "there is expression" is the main reason for this proposal,
> symbolic operator is not necessary.
closes a gap in the language. Just like list comprehensions fit generator
expressions and vice versa. There is no "need" for being able to write
[x**2 for x in seq]
{x**2 for x in seq}
when you can equally well write
list(x**2 for x in seq)
set(x**2 for x in seq)
But I certainly wouldn't complain about that redundancy in the language.
> `new = d1.updated(d2)` or `new = dict.merge(d1, d2)` are enough. Python
> preferred name over symbol in general. Symbols are readable and
> understandable only when it has good math metaphor.
>
> Sets has symbol operator because it is well known in set in math, not
> because set is frequently used.
>
> In case of dict, there is no simple metaphor in math.
operator, would you also reject the idea of adding it, argueing that we
could use this:
L = L1.extended(L2)
I honestly do not see the math relation in concatenation via "+".
But, given that "+" and "|" already have the meaning of "merging two
containers into one" in Python, I think it makes sense to allow that also
for dicts.
> It just cryptic and hard to Google.
more than they have to search for the list "+" operation. My guess is that
it's pretty much what most people would try first when they have the need
to merge two dicts, and only failing that, they would start a web search.
In comparison, very few users would be able to come up with "{**d1, **d2}"
on their own, or even "d1.updated(d2)".
My point is, given the current language, "dict+dict" is a gap that is worth
closing.
Stefan
Brice Parent
Le 05/03/2019 à 23:40, Greg Ewing a écrit :
> Steven D'Aprano wrote:
>> The question is, is [recursive merge] behaviour useful enough and
> > common enough to be built into dict itself?
>
> I think not. It seems like just one possible way of merging
> values out of many. I think it would be better to provide
> a merge function or method that lets you specify a function
> for merging values.
>
for the most general usage (and current PEP's describes the behaviour I
would expect before reading the doc), but for all other more specific
usages, where we intend any special or not-so-common behaviour, I'd go
with modifying Dict.update like this:
foo.update(bar, on_collision=updator) # Although I'm not a fan of the
keyword I used
`updator` being a simple function like this one:
def updator(updated, updator, key) -> Any:
if key == "related":
return updated[key].update(updator[key])
if key == "tags":
return updated[key] + updator[key]
if key in ["a", "b", "c"]: # Those
return updated[key]
return updator[key]
There's nothing here that couldn't be made today by using a custom
update function, but leaving the burden of checking for values that are
in both and actually inserting the new values to Python's language, and
keeping on our side only the parts that are specific to our use case,
makes in my opinion the code more readable, with fewer possible bugs and
possibly better optimization.
Inada Naoki
>
> INADA Naoki schrieb am 05.03.19 um 08:03:> On Tue, Mar 5, 2019 at 12:02 AM
> Stefan Behnel wrote:
> >> INADA Naoki schrieb am 04.03.19 um 11:15:
> >>> Why statement is not enough?
> >>
> >> I'm not sure I understand why you're asking this, but a statement is
> >> "not enough" because it's a statement and not an expression. It does
> >> not replace the convenience of an expression.
> >
> > It seems tautology and say nothing.
>
> That's close to what I thought when I read your question. :)
>
>
> > What is "convenience of an expression"?
>
> It's the convenience of being able to write an expression that generates
> the thing you need, rather than having to split code into statements that
> create it step by step before you can use it.
>
I don't think it's reasonable rationale for adding operator.
First, Python sometimes force people to use statement intentionally.
Strictly speaking, dict.update() is an expression. But it not return `self`
so you must split statements. It's design decision.
So "add operator because I want expression" is bad reasoning to me.
If it is valid reasoning, every mutating method should have operator.
It's crazy idea.
Second, operator is not required for expression. And adding operator
must have high bar than adding method because it introduces more
complexity and it could seen cryptic especially when the operator doesn't
have good math metaphor.
So I proposed adding dict.merge() instead of adding dict + as a counter
proposal.
If "I want expression" is really main motivation, it must be enough.
> Think of comprehensions versus for-loops. Comprehensions are expressions
> that don't add anything to the language that a for-loop cannot achieve.
> Still, everyone uses them because they are extremely convenient.
>
I agree that comprehension is extremely convenient.
But I think the main reason is it is compact and readable.
If comprehension is not compact and readable as for-loop, it's not extremely
convenient.
>
> > Is it needed to make Python more readable language?
>
> No, just like comprehensions, it's not "needed". It's just convenient.
>
I think comprehension is needed to make Python more readable language,
not just for convenient.
>
> > Anyway, If "there is expression" is the main reason for this proposal,
> > symbolic operator is not necessary.
>
> As said, "needed" is not the right word.
Maybe, I misunderstood nuance of the word "needed". English and Japanese
are very different language. sorry.
> Being able to use a decorator
> closes a gap in the language. Just like list comprehensions fit generator
> expressions and vice versa. There is no "need" for being able to write
>
> [x**2 for x in seq]
> {x**2 for x in seq}
>
> when you can equally well write
>
> list(x**2 for x in seq)
> set(x**2 for x in seq)
>
> But I certainly wouldn't complain about that redundancy in the language.
>
OK, I must agree this point. [] and {} has good metaphor in math.
We use [1, 2, 3,... ] for series, and {1, 2, 3, ...} for sets.
>
> > `new = d1.updated(d2)` or `new = dict.merge(d1, d2)` are enough. Python
> > preferred name over symbol in general. Symbols are readable and
> > understandable only when it has good math metaphor.
> >
> > Sets has symbol operator because it is well known in set in math, not
> > because set is frequently used.
> >
> > In case of dict, there is no simple metaphor in math.
>
> So then, if "list+list" and "tuple+tuple" wasn't available through an
> operator, would you also reject the idea of adding it, argueing that we
> could use this:
>
> L = L1.extended(L2)
>
> I honestly do not see the math relation in concatenation via "+".
>
First of all, concatenating sequence (especially str) is extremely
frequent than merging dict. My point is dict + dict is major abuse
of + than seq + seq and it's usage is smaller than seq + seq.
Let's describe why I think dict+dict is "major" abuse.
As I said before, it's common to assign operator for concatenation in
regular language, while middle-dot is used common.
When the commonly-used operator is not in ASCII, other symbol can
be used as alternative. We used | instead of ∪.
In case of dict, it's not common to assign operator for merging in math,
as far as I know.
(Maybe, "direct sum" ⊕ is similar to it. But it doesn't allow intersection.
So ValueError must be raised for duplicated key if we use "direct sum"
for metaphor.
But direct sum is higher-level math than "union" of set. I don't think
it's good idea to use it as metaphor.)
That's one of reasons I think seq + seq is "little" abuse and dict +
dict is "major" abuse.
Another reason is "throw some values away" doesn't fit mental model of "sum",
as I said already in earlier mail.
> But, given that "+" and "|" already have the meaning of "merging two
> containers into one" in Python, I think it makes sense to allow that also
> for dicts.
>
+ is used for concatenate, it is more strict than just merge.
If + is allowed for dict, set should support it too for consistency.
Then, meaning of "+ for container" become "sum up two containers in
some way, defined by the container type." It's consistent.
Kotlin uses + for this meaning. Scala uses ++ for this meaning.
But this is a large design change of the language. Is this really required?
I feel adding a method is enough.
--
Inada Naoki <songof...@gmail.com>
Rémi Lapeyre
(con...@brice.xyz(mailto:con...@brice.xyz)) a écrit:
>
> Le 05/03/2019 à 23:40, Greg Ewing a écrit :
> > Steven D'Aprano wrote:
> >> The question is, is [recursive merge] behaviour useful enough and
> > > common enough to be built into dict itself?
> >
> > I think not. It seems like just one possible way of merging
> > values out of many. I think it would be better to provide
> > a merge function or method that lets you specify a function
> > for merging values.
> >
> That's what this conversation led me to. I'm not against the addition
> for the most general usage (and current PEP's describes the behaviour I
> would expect before reading the doc), but for all other more specific
> usages, where we intend any special or not-so-common behaviour, I'd go
> with modifying Dict.update like this:
>
> foo.update(bar, on_collision=updator) # Although I'm not a fan of the
> keyword I used
Le 6 mars 2019 à 10:26:15, Brice Parent
(con...@brice.xyz(mailto:con...@brice.xyz)) a écrit:
>
> Le 05/03/2019 à 23:40, Greg Ewing a écrit :
> > Steven D'Aprano wrote:
> >> The question is, is [recursive merge] behaviour useful enough and
> > > common enough to be built into dict itself?
> >
> > I think not. It seems like just one possible way of merging
> > values out of many. I think it would be better to provide
> > a merge function or method that lets you specify a function
> > for merging values.
> >
> That's what this conversation led me to. I'm not against the addition
> for the most general usage (and current PEP's describes the behaviour I
> would expect before reading the doc), but for all other more specific
> usages, where we intend any special or not-so-common behaviour, I'd go
> with modifying Dict.update like this:
>
> foo.update(bar, on_collision=updator) # Although I'm not a fan of the
> keyword I used
This won’t be possible update() already takes keyword arguments:
>>> foo = {}
>>> bar = {'a': 1}
>>> foo.update(bar, on_collision=lambda e: e)
>>> foo
{'a': 1, 'on_collision': <function <lambda> at 0x10b8df598>}
Ka-Ping Yee
Rhodri James
> len(dict1 + dict2) does not equal len(dict1) + len(dict2), so using the +
> operator is nonsense.
important. Making assumptions about length where any dictionary
manipulations are concerned seems unwise to me, which makes a nonsense
of your claim that this is nonsense :-)
--
Rhodri James *-* Kynesim Ltd
Brice Parent
Le 06/03/2019 à 10:50, Rémi Lapeyre a écrit :
>> Le 05/03/2019 à 23:40, Greg Ewing a écrit :
>>> Steven D'Aprano wrote:
>>>> The question is, is [recursive merge] behaviour useful enough and
>>>> common enough to be built into dict itself?
>>> I think not. It seems like just one possible way of merging
>>> values out of many. I think it would be better to provide
>>> a merge function or method that lets you specify a function
>>> for merging values.
>>>
>> That's what this conversation led me to. I'm not against the addition
>> for the most general usage (and current PEP's describes the behaviour I
>> would expect before reading the doc), but for all other more specific
>> usages, where we intend any special or not-so-common behaviour, I'd go
>> with modifying Dict.update like this:
>>
>> foo.update(bar, on_collision=updator) # Although I'm not a fan of the
>> keyword I used
> This won’t be possible update() already takes keyword arguments:
>
>>>> foo = {}
>>>> bar = {'a': 1}
>>>> foo.update(bar, on_collision=lambda e: e)
>>>> foo
> {'a': 1, 'on_collision': <function <lambda> at 0x10b8df598>}
I don't see that as a problem at all.
Having a function's signature containing a **kwargs doesn't disable to
have explicit keyword arguments at the same time:
`def foo(bar="baz", **kwargs):` is perfectly valid, as well as `def
spam(ham: Dict, eggs="blah", **kwargs):`, so `update(other,
on_collision=None, **added) is too, no? The major implication to such a
modification of the Dict.update method, is that when you're using it
with keyword arguments (by opposition to passing another dict/iterable
as positional), you're making a small non-backward compatible change in
that if in some code, someone was already using the keyword that would
be chosing (here "on_collision"), their code would be broken by the new
feature.
I had never tried to pass a dict and kw arguments together, as it seemed
to me that it wasn't supported (I would even have expected an exception
to be raised), but it's probably my level of English that isn't high
enough to get it right, or this part of the doc that doesn't describe
well the full possible usage of the method (see here:
https://docs.python.org/3/library/stdtypes.html#dict.update). Anyway, if
the keyword is slected wisely, the collision case will almost never
happen, and be quite easy to correct if it ever happened.
Jonathan Fine
>
> len(dict1 + dict2) does not equal len(dict1) + len(dict2), so using the + operator is nonsense.
>
> len(dict1 + dict2) cannot even be computed by any expression involving +. Using len() to test the semantics of the operation is not arbitrary; the fact that the sizes do not add is a defining quality of a merge. This is a merge, not an addition. The proper analogy is to sets, not lists.
concern about this proposal. I think most us will agree that the
proposal is to use '+' to express a merge operation, namely update.
(There are other merge operations, when there are two values to
combine, such as taking the min or max of the two values.)
Certainly, many of the posts quite naturally use the word merge.
Indeed PEP 584 writes "This PEP suggests adding merge '+' and
difference '-' operators to the built-in dict class."
We would all agree that it would be obviously wrong to suggest adding
merge '-' and difference '+' operators. (Note: I've swapped '+' and
'-'.) And why? Because it is obviously wrong to use '-' to denote
merge, etc.
Some of us are also upset by the use of '+' to denote merge. By the
way, there is already a widespread symbol for merge. It appears on
many road signs. It looks like an upside down 'Y'. It even has merge
left and merge right versions.
Python already has operator symbols '+', '-', '*', '/' and so on. See
https://docs.python.org/3/reference/lexical_analysis.html#operators
Perhaps we should add a merge or update symbol to this list, so that
we don't overload to breaking point the humble '+' operator. Although
that would make Python a bit more like APL.
By the way, Pandas already has a merge operation, called merge, that
takes many parameters. I've only glanced at it.
https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.merge.html
--
Jonathan
Josh Rosenberg
On 06/03/2019 10:29, Ka-Ping Yee wrote:
> len(dict1 + dict2) does not equal len(dict1) + len(dict2), so using the +
> operator is nonsense.
I'm sorry, but you're going to have to justify why this identity is
important. Making assumptions about length where any dictionary
manipulations are concerned seems unwise to me, which makes a nonsense
of your claim that this is nonsense :-)
Jonathan Fine
> Making assumptions about length where any dictionary
> manipulations are concerned seems unwise to me
a = len(d1)
b = len(d2)
c = len(d1 + d2) # Using the suggested syntax.
Then we know
max(a, b) <= c <= a + b
And this is, in broad terms, characteristic of merge operations.
--
Jonathan
David Mertz
Chris Angelico
> The major implication to such a
> modification of the Dict.update method, is that when you're using it
> with keyword arguments (by opposition to passing another dict/iterable
> as positional), you're making a small non-backward compatible change in
> that if in some code, someone was already using the keyword that would
> be chosing (here "on_collision"), their code would be broken by the new
> feature.
> the keyword is slected wisely, the collision case will almost never
> happen, and be quite easy to correct if it ever happened.
Let's suppose that someone had indeed used the chosen keyword (and
remember, the more descriptive the argument name, the more likely that
it'll be useful elsewhere and therefore have a collision). How would
they discover this? If they're really lucky, there MIGHT be an
exception (if on_collision accepts only a handful of keywords, and the
collision isn't one of them), but if your new feature is sufficiently
flexible, that might not happen. There'll just be incorrect behaviour.
As APIs go, using specific keyword args at the same time as **kw is a
bit odd. Consider:
button_options.update(button_info, on_click=frobnicate, style="KDE",
on_collision="replace")
It's definitely not obvious which of those will end up in the
dictionary and which won't. Big -1 from me on that change.
ChrisA
Brice Parent
That's indeed a good point. Even if the correction is quite easy to make
in most cases. With keyword only changes:
button_options.update(dict(on_click=frobnicate, style="KDE", on_collision="replace")) # or button_options.update(dict(on_collision="replace"), on_click=frobnicate, style="KDE")
In the exact case you proposed, it could become a 2-liners:
button_options.update(button_info)
button_options.update(dict(on_click=frobnicate, style="KDE", on_collision="replace"))
In my code, I would probably make it into 2 lines, to make clear that we
have 2 levels of data merging, one that is general (the first), and one
that is specific to this use-case (as it's hard written in the code),
but not everyone doesn't care about the number of lines.
But for the other part of your message, I 100% agree with you. The main
problem with such a change is not (to me) that it can break some edge
cases, but that it would potentially break them silently. And that, I
agree, is worth a big -1 I guess.
Michael Lee
I strongly agree with Ka-Ping. '+' is intuitively concatenation not merging. The behavior is overwhelmingly more similar to the '|' operator in sets (whether or not a user happens to know the historical implementation overlap).
Chris Angelico
> If your instinct is to assume "+" means "concatenation", then it would be natural to assume that {"a": 1, "b": 2} + {"c": 3, "b": 4} would be identical to {"a": 1, "b": 2, "c": 3, "b": 4} -- literally concat the key-value pairs into a new dict.
>
> But of course, you can't have duplicate keys in Python. So, you would either recall or look up how duplicate keys are handled when constructing a dict and learn that the rule is that the right-most key wins. So the natural conclusion is that "+" would follow this existing rule -- and you end up with exactly the behavior described in the PEP.
>
d = {}; d["a"] = 1; d["b"] = 2; d["c"] = 3; d["b"] = 4
Rightmost one wins. It's the most logical behaviour.
ChrisA
Jonathan Fine
> If your instinct is to assume "+" means "concatenation", then it would be natural to assume that {"a": 1, "b": 2} + {"c": 3, "b": 4} would be identical to {"a": 1, "b": 2, "c": 3, "b": 4} -- literally concat the key-value pairs into a new dict.
> But of course, you can't have duplicate keys in Python. So, you would either recall or look up how duplicate keys are handled when constructing a dict and learn that the rule is that the right-most key wins. So the natural conclusion is that "+" would follow this existing rule -- and you end up with exactly the behavior described in the PEP.
that taught me something. Here goes:
>>> {'a': 0}
{'a': 0}
>>> {'a': 0, 'a': 0}
{'a': 0}
>>> {'a': 0, 'a': 1}
{'a': 1}
>>> {'a': 1, 'a': 0}
{'a': 0}
This surprised me quite a bit. I was expecting to get an exception. However
>>> dict(a=0)
{'a': 0}
>>> dict(a=0, a=0)
SyntaxError: keyword argument repeated
does give an exception.
I wonder, is this behaviour of {'a': 0, 'a': 1} documented (or tested)
anywhere? I didn't find it in these URLs:
https://docs.python.org/3/library/stdtypes.html#mapping-types-dict
https://docs.python.org/3/tutorial/datastructures.html#dictionaries
I think this behaviour might give rise to gotchas. For example, if we
define inverse_f by
>>> inverse_f = { f(a): a, f(b): b }
then is the next statement always true (assuming a <> b)?
>>> inverse_f[ f(a) ] == a
Well, it's not true with these values
>>> a, b = 1, 2
>>> def f(n): pass # There's a bug here, f(n) should be a bijection.
A quick check that len(inverse) == 2 would provide a sanity check. Or
perhaps better, len(inverse_f) == len(set(a, b)). (I don't have an
example of this bug appearing 'in the wild'.)
Once again, I thank Michael for his nice, instructive and
well-presented example.
--
Jonathan
Inada Naoki
>
> I think the behavior proposed in the PEP makes sense whether you think of "+" as meaning "concatenation" or "merging".
>
> If your instinct is to assume "+" means "concatenation", then it would be natural to assume that {"a": 1, "b": 2} + {"c": 3, "b": 4} would be identical to {"a": 1, "b": 2, "c": 3, "b": 4} -- literally concat the key-value pairs into a new dict.
>
Better example, {"a": 1, "b": 2} + {"c": 4, "b": 3} == {"a": 1, "b":
2, "c": 4, "b": 3} == {"a": 1, "b": 3, "c": 4}
On the other hand, union of set is also "literally concat". If we use
this "literally concat" metaphor,
I still think set should have `+` as alias to `|` for consistency.
>
> Using "|" would also violate an important existing property of unions: the invariant "d1 | d2 == d2 | d1" is no longer true. As far as I'm aware, the union operation is always taken to be commutative in math, and so I think it's important that we preserve that property in Python. At the very least, I think it's far more important to preserve commutativity of unions then it is to preserve some of the invariants I've seen proposed above, like "len(d1 + d2) == len(d1) + len(d2)".
>
I think "|" keeps commutativity only because it's minor than `+`. Easy operator
is abused easily more than minor operator.
And I think every "coincidence" rules are important. They makes
understanding Python easy.
Every people "discover" rules and consistency while learning language.
This is a matter of balance. There are no right answer. Someone
*feel* rule A is important than B.
Someone feel opposite.
> But I do know that I'm a strong -1 on adding set operations to dicts: it's not possible to preserve the existing semantics of union (and intersection) with dict and think expressions like "d1 | d2" and "d1 & d2" would just be confusing and misleading to encounter in the wild.
To me, {"a": 1, "b": 2} - {"b": 3} = {"a": 1} is confusing than {"a":
1, "b": 2} - {"b"} = {"a": 1}.
So I think borrow some semantics from set is good idea.
Both of `dict - set` and `dict & set` makes sense to me.
* `dict - set` can be used to remove private keys by "blacklist".
* `dict & set` can be used to choose public keys by "whiltelist".
--
Inada Naoki <songof...@gmail.com>
Jonathan Fine
keys. No conclusions (so far).
BACKGROUND
In the thread "PEP: Dict addition and subtraction" I wrote
> >>> {'a': 0, 'a': 1}
> {'a': 1}
> anywhere? I didn't find it in these URLs:
> https://docs.python.org/3/library/stdtypes.html#mapping-types-dict
> https://docs.python.org/3/tutorial/datastructures.html#dictionaries
I've since found some relevant URLs.
[1] https://stackoverflow.com/questions/34539772/is-a-dict-literal-containing-repeated-keys-well-defined
[2] https://help.semmle.com/wiki/display/PYTHON/Duplicate+key+in+dict+literal
[3] https://bugs.python.org/issue26910
[4] https://bugs.python.org/issue16385
[5] https://realpython.com/python-dicts/
ANALYSIS
[1] gives a reference to [6], which correctly states the behaviour of
{'a':0, 'a':1}, although without giving an example. (Aside: Sometimes
one example is worth 50 or more words.)
[2] is from Semmle, who provide an automated code review tool, called
LGTM. The page [2] appears to be part of the documentation for LGTM.
This page provides a useful link to [7].
[3] is a re-opening of [4]. It was rapidly closed by David Murray, who
recommended reopening the discussion on python-ideas.
[4] was raised by Albert Ferras, based on his real-world experience.
In particular, a configuration file that contains a long dict literal.
This was closed by Benjamin Peterson, who said that raising an error
was "out of the question for compatibility isssues". Given few use
case and little support on python-ideas,Terry Ready supported the
closure. Raymond Hettinger supported the closure.
[5] is from RealPython, who provide online tutorials. This page
contains the statement "a given key can appear in a dictionary only
once. Duplicate keys are not allowed." Note that
{'a': 0, 'a': 1}
can reasonably be thought of as a dictionary with duplicate keys.
NOTE
As I recall SGML (this shows my age) allows multiple entity declarations, as in
<!ENTITY key "original">
<!ENTITY key "updated">
And as I recall, in SGML the first value "original" is the one that is
in effect. This is what happens with the LaTeX command
\providecommand.
FURTHER LINKS
[6] https://docs.python.org/3/reference/expressions.html#dictionary-displays
[7] https://cwe.mitre.org/data/definitions/561.html # CWE-561: Dead Code
Jonathan Fine
> I wonder, is this behaviour of {'a': 0, 'a': 1} documented (or tested)
> anywhere?
[Python-ideas] dict literal allows duplicate keys
https://mail.python.org/pipermail/python-ideas/2019-March/055717.html
Finally, Christopher Barker wrote:
> Yes, and had already been brought up in this thread ( I think by Guido). (Maybe not well documented, but certainly well understood and deliberate)
Thank you for this, Christopher.
Inada Naoki
> If a comma-separated sequence of key/datum pairs is given, they are evaluated from left to right to define the entries of the dictionary: each key object is used as a key into the dictionary to store the corresponding datum. This means that you can specify the same key multiple times in the key/datum list, and the final dictionary’s value for that key will be the last one given.
--
Inada Naoki <songof...@gmail.com>
Jonathan Fine
Inada Naoki quoted (from doc.python ref [6] in my original post):
> > If a comma-separated sequence of key/datum pairs is given, they are evaluated from left to right to define the entries of the dictionary: each key object is used as a key into the dictionary to store the corresponding datum. This means that you can specify the same key multiple times in the key/datum list, and the final dictionary’s value for that key will be the last one given.
Indeed. Although off-topic, I think
>>> {'a': 0, 'a': 1} == {'a': 1}
True
is much better than "This means that you can specify the same key
multiple times in the key/datum list, and the final dictionary’s value
for that key will be the last one given."
By the way, today I think we'd say key/value pairs. And I've read
https://www.theguardian.com/guardian-observer-style-guide-d
data takes a singular verb (like agenda), though strictly a plural;
you come across datum, the singular of data, about as often as you
hear about an agendum
Oh, and "the final dictionary's value" should I think be "the
dictionary's final value" or perhaps just "the dictionary's value"
But now we're far from the thread topic. I'm happy to join in on a
thread on improving documentation (by using simpler language and good
examples).
Rhodri James
> Indeed. Although off-topic, I think
>
>>>> {'a': 0, 'a': 1} == {'a': 1}
> True
>
> is much better than "This means that you can specify the same key
> multiple times in the key/datum list, and the final dictionary’s value
> for that key will be the last one given."
I disagree. An example is an excellent thing, but the words are
definitive and must be there.
--
Rhodri James *-* Kynesim Ltd
Rhodri James
> On 06/03/2019 17:43, Jonathan Fine wrote:
>> Indeed. Although off-topic, I think
>>
>>>>> {'a': 0, 'a': 1} == {'a': 1}
>> True
>>
>> is much better than "This means that you can specify the same key
>> multiple times in the key/datum list, and the final dictionary’s value
>> for that key will be the last one given."
>
> I disagree. An example is an excellent thing, but the words are
> definitive and must be there.
Sigh. I hit SEND before I finished changing the title. Sorry, folks.
Michael Lee
If we use this "literally concat" metaphor, I still think set should have `+` as alias to `|` for consistency.
I think "|" keeps commutativity only because it's minor than `+`.
Hmm. The PEP proposed dict - dict, which is similar to set - set (difference).
Guido van Rossum
Jonathan Fine
You wrote:
> Would it shut down this particular subthread if (as the language's designer, if not its BDFL) I declared that this was an explicit design decision that I made nearly 30 years ago? I should perhaps blog about the background of this decision, but it was quite a conscious one. There really is no point in thinking that this is an accident of implementation or could be changed.
I'd be fascinated to hear about the background to this conscious
decision, and I think it would help me and others understand better
what makes Python what it is. And it might help persuade me that my
surprise at {'a': 0, 'a': 1} is misplaced, or at least exaggerated and
one-sided.
Do you want menial help writing the blog? Perhaps if you share your
recollections, others will find the traces in the source code. For
example, I've found the first dictobject.c, dating back to 1994.
https://github.com/python/cpython/blob/956640880da20c20d5320477a0dcaf2026bd9426/Objects/dictobject.c
I'm a great fan of your Python conversation (with Biancuzzi and Warden) in
http://shop.oreilly.com/product/9780596515171.do # Masterminds of Programming
I've read this article several times, and have wished that it was more
widely available. My personal view is that putting a copy of this
article in docs.python.org would provide more benefit to the community
than you blogging on why dict literals allow duplicate keys. However,
it need not be either/or. Perhaps someone could ask the PSF to talk
with O'Reilly about getting copyright clearance to do this.
Finally, some personal remarks. I've got a long training as a pure
mathematician. For me consistency and application of simple basic
principles is important to me. And also the discovery of basic
principles.
In your interview, you say (paraphrased and without context) that most
Python code is written simply to get a job done. And that pragmatism,
rather then being hung up about theoretical concept, is the
fundamental quality in being proficient in developing with Python.
Thank you for inventing Python, and designing the language. It's a
language popular both with pure mathematicians, and also pragmatic
people who want to get things done. That's quite an achievement, which
has drawn people like me into your community.
with best regards
Greg Ewing
> len(dict1 + dict2) does not equal len(dict1) + len(dict2), so using the
> + operator is nonsense.
nonsense, because len(v1 + v2) is not in general equal to
len(v1) + len(v2).
Yet mathematicians are quite happy to talk about "addition"
of vectors.
--
Greg
Josh Rosenberg
You might as well say that using the + operator on vectors is
nonsense, because len(v1 + v2) is not in general equal to
len(v1) + len(v2).
Yet mathematicians are quite happy to talk about "addition"
of vectors.
Chris Angelico
<shadowranger...@gmail.com> wrote:
> The closest I can come to a thorough definition of what + does in Python (and most languages) right now is that:
>
> 1. Returns a new thing of the same type (or a shared coerced type for number weirdness)
> 2. That combines the information of the input operands
> 3. Is associative ((a + b) + c produces the same thing as a + (b + c)) (modulo floating point weirdness)
> 4. Is "reversible": Knowing the end result and *one* of the inputs is sufficient to determine the value of the other input; that is, for c = a + b, knowing any two of a, b and c allows you to determine a single unambiguous value for the remaining value (numeric coercion and floating point weirdness make this not 100%, but you can at least know a value equal to other value; e.g. for c = a + b, knowing c is 5.0 and a is 1.0 is sufficient to say that b is equal to 4, even if it's not necessarily an int or float). For numbers, reversal is done with -; for sequences, it's done by slicing c using the length of a or b to "subtract" the elements that came from a/b.
> 5. (Actual addition only) Is commutative (modulo floating point weirdness); a + b == b + a
> 6. (Concatenation only) Is order preserving (really a natural consequence of #4, but a property that people expect)
>
>
> 1. Fewer consistent rules apply to +;
> 2. The particular idiosyncrasies of Python dict ordering and "which value wins" rules are now tied to +. for concatenation, there is only one set of possible rules AFAICT so every language naturally agrees on behavior, but dict merging obviously has many possible rules that would be unlikely to match the exact rules of any other language except by coincidence). a winning on order and b winning on value is a historical artifact of how Python's dict developed; I doubt any other language would intentionally choose to split responsibility like that if they weren't handcuffed by history.
>
> Again, there's nothing wrong with making dict merges easier. But it shouldn't be done by (further) abusing +.
But adding dictionaries is fundamentally *useful*. It is expressive.
It will, in pretty much all situations, do exactly what someone would
expect, based on knowledge of how Python works in other areas. The
semantics for edge cases have to be clearly defined, but they'll only
come into play on rare occasions; most of the time, for instance, we
don't have to worry about identity vs equality in dictionary keys. If
you tell people "adding two dictionaries combines them, with the right
operand winning collisions", it won't matter that this isn't how lists
or floats work; it'll be incredibly useful as it is.
Practicality. Let's have some.
ChrisA
Ka-Ping Yee
On Thu, Mar 7, 2019 at 10:52 AM Josh Rosenberg
<shadowranger...@gmail.com> wrote:
>
> Allowing dicts to get involved in + means:
Lots of words that basically say: Stuff wouldn't be perfectly pure.
But adding dictionaries is fundamentally *useful*. It is expressive.
If we were inventing programming languages in a vacuum, you could say + can mean "arbitrary combination operator" and it would be fine. But we're not in a vacuum; every major language that uses + with general purpose containers uses it to mean element-wise addition or concatenation, not just "merge".
James Lu
d1 << d2 makes a copy of d1 and merges d2 into it, and when the keys conflict, d2 takes priority. (Works like copy/update.)
d1 + d2 makes a new dictionary, taking keys from d1 and d2. If d1 and d2 have a different value for same key, a KeyError is thrown.
Stephen J. Turnbull
> On Wed, Mar 6, 2019 at 4:01 PM Chris Angelico <ros...@gmail.com> wrote:
> > But adding dictionaries is fundamentally *useful*. It is expressive.
>
> It is useful. It's just that + is the wrong name.
apparent. I'm also aware that I have biases so I'm sympathetic to
those who take a different position.
Rather than say it's "wrong", let me instead point out that I think
it's pragmatically troublesome to use "+". I can think of at least
four interpretations of "d1 + d2"
1. update
2. multiset (~= Collections.Counter addition)
3. addition of functions into the same vector space (actually, a
semigroup will do ;-), and this is the implementation of
Collections.Counter
4. "fiberwise" addition (ie, assembling functions into relations)
and I'm very jet-lagged so I may be missing some.
Since "|" (especially "|=") *is* suitable for "update", I think we
should reserve "+" for some alternative future commutative extension,
of which there are several possible (all of 2, 3, 4 are commutative).
Again in the spirit of full disclosure, of those above, 2 is already
implemented and widely used, so we don't need to use "+" for that.
I've never seen 4 except in the mathematical literature (union of
relations is not the same thing). 3, however, is very common both for
mappings with small domain and sparse representation of mappings with
a default value (possibly computed then cached), and "|" is not
suitable for expressing that sort of addition (I'm willing to say it's
"wrong" :-).
There's also the fact that the operations denoted by "|" and "||" are
often implemented as "short-circuiting", and therefore not
commutative, while "+" usually is (and that's reinforced for
mathematicians who are trained to think of "+" as the operator for
Abelian groups, while "*" is a (possibly) non-commutative operator. I
know commutativity of "+" has been mentioned before, but the
non-commutativity of "|" -- and so unsuitability for many kinds of
dict combination -- hasn't been emphasized before IIRC.
Steve
Stephen J. Turnbull
> On Wed, Mar 6, 2019 at 4:01 PM Chris Angelico <ros...@gmail.com> wrote:
> > But adding dictionaries is fundamentally *useful*. It is expressive.
>
> It is useful. It's just that + is the wrong name.
apparent. I'm also aware that I have biases so I'm sympathetic to
those who take a different position.
Rather than say it's "wrong", let me instead point out that I think
it's pragmatically troublesome to use "+". I can think of at least
four interpretations of "d1 + d2"
1. update
2. multiset (~= Collections.Counter addition)
3. addition of functions into the same vector space (actually, a
semigroup will do ;-), and this is the implementation of
Collections.Counter
and I'm very jet-lagged so I may be missing some.
often implemented as "short-circuiting", and therefore not
commutative, while "+" usually is (and that's reinforced for
mathematicians who are trained to think of "+" as the operator for
Abelian groups, while "*" is a (possibly) non-commutative operator. I
know commutativity of "+" has been mentioned before, but the
non-commutativity of "|" -- and so unsuitability for many kinds of
dict combination -- hasn't been emphasized before IIRC.
In the spirit of full disclosure:
Of these, 2 is already implemented and widely used, so we don't need
to use dict.__add__ for that. I've never seen 4 in the mathematical
very common both for mappings with small domain and sparse
representation of mappings with a default value (possibly computed
then cached), and "|" is not suitable for expressing that sort of
addition (I'm willing to say it's "wrong" :-).
João Matos
I've just read your PEP 585 draft and have some questions.
When you say
"
Like the merge operator and list concatenation, the difference operator requires both operands to be dicts, while the augmented version allows any iterable of keys.
">>> d - {'spam', 'parrot'} Traceback (most recent call last): ... TypeError: cannot take the difference of dict and set>>> d -= {'spam', 'parrot'} >>> print(d) {'eggs': 2, 'cheese': 'cheddar'}>>> d -= [('spam', 999)] >>> print(d) {'spam': 999, 'eggs': 2, 'cheese': 'cheddar', 'aardvark': 'Ethel'}
The option d -= {'spam', 'parrot'} where parrot does not exist in the d dict, will raise an exception (eg. KeyNotFound) or be silent?
The option d -= [('spam', 999)] should remove the pair from the dict, correct? But the print that follows still shows it there. It's a mistake or am I missing something?
Best regards,
João Matos
Guido van Rossum
Ka-Ping Yee writes:
> On Wed, Mar 6, 2019 at 4:01 PM Chris Angelico <ros...@gmail.com> wrote:
> > But adding dictionaries is fundamentally *useful*. It is expressive.
>
> It is useful. It's just that + is the wrong name.
First, let me say that I prefer ?!'s position here, so my bias is made
apparent. I'm also aware that I have biases so I'm sympathetic to
those who take a different position.
Rather than say it's "wrong", let me instead point out that I think
it's pragmatically troublesome to use "+". I can think of at least
four interpretations of "d1 + d2"
1. update
2. multiset (~= Collections.Counter addition)
3. addition of functions into the same vector space (actually, a
semigroup will do ;-), and this is the implementation of
Collections.Counter
4. "fiberwise" set addition (ie, of functions into relations)
and I'm very jet-lagged so I may be missing some.
There's also the fact that the operations denoted by "|" and "||" are
often implemented as "short-circuiting", and therefore not
commutative, while "+" usually is (and that's reinforced for
mathematicians who are trained to think of "+" as the operator for
Abelian groups, while "*" is a (possibly) non-commutative operator. I
know commutativity of "+" has been mentioned before, but the
non-commutativity of "|" -- and so unsuitability for many kinds of
dict combination -- hasn't been emphasized before IIRC.
Since "|" (especially "|=") *is* suitable for "update", I think we
should reserve "+" for some future commutative extension.
In the spirit of full disclosure:
Of these, 2 is already implemented and widely used, so we don't need
to use dict.__add__ for that. I've never seen 4 in the mathematical
literature (union of relations is not the same thing). 3, however, is
very common both for mappings with small domain and sparse
representation of mappings with a default value (possibly computed
then cached), and "|" is not suitable for expressing that sort of
addition (I'm willing to say it's "wrong" :-).
MRAB
[snip]
> mutual distributive properties will hold.
>
> Since "|" (especially "|=") *is* suitable for "update", I think we
> should reserve "+" for some future commutative extension.
>
>
> One argument is that sets have an update() method aliased to "|=", so
> this makes it more reasonable to do the same for dicts, which also have
> a. update() method, with similar behavior (not surprising, since sets
> were modeled after dicts).
>
One way to think of it is that a dict is like a set, except that each of
its members has an additional associated value.
Greg Ewing
> I guess this explains the behavior of removing results <= 0; it makes
> sense as multiset subtraction, since in a multiset a negative count
> makes little sense. (Though the name Counter certainly doesn't seem to
> imply multiset.)
is happy to let the value go negative.
> For sets,
> union and intersection are distributive over each other.
> for union and intersection got the operators wrong.
But in another sense, it didn't. In Boolean algebra, "and" and "or"
(which also distribute over each other) are often written using the
same notations as multiplication and addition. There's no rule in
mathematics saying that these notations must be distributive in one
direction but not the other.
--
Greg
Guido van Rossum
Guido van Rossum wrote:
> I guess this explains the behavior of removing results <= 0; it makes
> sense as multiset subtraction, since in a multiset a negative count
> makes little sense. (Though the name Counter certainly doesn't seem to
> imply multiset.)
It doesn't even behave consistently as a multiset, since c[k] -= n
is happy to let the value go negative.
> For sets,
> union and intersection are distributive over each other.
> Note that this is *not* the case for + and * when used with
> (mathematical) numbers... So in a sense, SETL (which uses + and *
> for union and intersection got the operators wrong.
But in another sense, it didn't. In Boolean algebra, "and" and "or"
(which also distribute over each other) are often written using the
same notations as multiplication and addition. There's no rule in
mathematics saying that these notations must be distributive in one
direction but not the other.
Greg Ewing
> I guess everybody's high school math(s) class was different. I don't
> ever recall seeing + and * for boolean OR/AND; we used ∧ and ∨.
Boolean algebra was only touched on briefly in my high school
years. I can't remember exactly what notation was used, but it
definitely wasn't ∧ and ∨ -- I didn't encounter those until
much later.
However, I've definitely seen texts on boolean alegbra in
relation to logic circuits that write 'A and B' as 'AB',
and 'A or B' as 'A + B'. (And also use an overbar for
negation instead of the mathematical ¬).
Maybe it depends on whether you're a mathematician or an
engineer? The multiplication-addition notation seems a lot
more readable when you have a complicated boolean expression,
so I can imagine it being favoured by pragmatic engineering
type people.
James Edwards
Steven D'Aprano
reading all the comments even if I haven't responded.
I'm currently working on an update to the PEP which will, I hope,
improve some of the failings of the current draft.
--
Steven
Stephan Hoyer
This would be consistent with other infix operations on mutable ABCs, but could potentially break backwards compatibility for anyone who has defined a MutableMapping subclass that implements __add__ but not __iadd__.
Steven D'Aprano
> Would __iadd__ and __isub__ be added to collections.abc.MutableMapping?
dict itself. If people want to add this to MutableMapping, that could be
considered seperately.
Chris Barker via Python-ideas
Christopher Barker, Ph.D.
Oceanographer
Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception
Chris....@noaa.gov
Antoine Pitrou
Josh Rosenberg
<shadowranger...@gmail.com>
wrote:
>
> Overloading + lacks the clear descriptive aspect of update that describes
> the goal of the operation, and contradicts conventions (in Python and
> elsewhere) about how + works (addition or concatenation, and a lot of
> people don't even like it doing the latter, though I'm not that pedantic).
>
> A couple "rules" from C++ on overloading are "*Whenever the meaning of an
> operator is not obviously clear and undisputed, it should not be
> overloaded.* *Instead, provide a function with a well-chosen name.*"
> and "*Always
> stick to the operator’s well-known semantics".* (Source:
> https://stackoverflow.com/a/4421708/364696 , though the principle is
> restated in many other places).
Agreed with this. What is so useful exactly in this new dict operator
that it hasn't been implemented, say, 20 years ago? I rarely find
myself merging dicts and, when I do, calling dict.update() is entirely
acceptable (I think the "{**d}" notation was already a mistake, making
a perfectly readable operation more cryptic simply for the sake of
saving a few keystrokes).
Built-in operations should be added with regard to actual user needs
(such as: a first-class notation for matrix multiplication, making
formulas easier to read and understand), not a mere "hmm this might
sometimes be useful".
Besides, if I have two dicts with e.g. lists as values, I *really*
dislike the fact that the + operator will clobber the values rather than
concatenate them. It's a recipe for confusion.
Regards
Antoine.
Antoine Pitrou
Stefan Behnel <stef...@behnel.de> wrote:
> INADA Naoki schrieb am 04.03.19 um 11:15:
> > Why statement is not enough?
>
> I'm not sure I understand why you're asking this, but a statement is "not
> enough" because it's a statement and not an expression.
This is an argument for Perl 6, not for Python.