python 3 dict: .keys(), .values(), and .item()

[Ethan Furman]

In Python 2 we have:

dict().keys() \
dict().items() --> separate list() of the results
dict().values() /

and

dict().iter_keys() \
dict().iter_items() --> integrated iter() of the results
dict().iter_values() /

By "separate list" I mean a snapshot of the dict at the time, and by
"integrated iter()" I mean changes to the dict during iteration are
seen by the iter.

In Python 3 the iter_* methods replaced the list() type methods, which
makes sense from the point-of-view of moving to a more iterator based
language; however, as a result of that change the typical "iterate over
a dict" operation now has a built-in gotcha: modifying the dict during
the iteration can now cause exceptions.

The solution, of course, is simple: surround the iterator call with list():

list(dict.keys())
list(dict.items())
list(dict.values())

for k, v in list(flag._value2member_map_.items()):
...

The solution, however, feels a lot more boilerplate-ish. Either the
programmer takes a lot more care to remember the current state of the dict
(since it's no longer a snapshot), or "list()" is sprinkled around every
iterator access.

In other words, what used to be a completely safe operation now is not.

Thoughts?

--
~Ethan~

[Peter Otten]

Ethan Furman wrote:

> In Python 2 we have:
>
> dict().keys() \
> dict().items() --> separate list() of the results
> dict().values() /
>
> and
>
> dict().iter_keys() \
> dict().iter_items() --> integrated iter() of the results
> dict().iter_values() /

I guess you didn't use these as often as I did ;)

By the way, there are also the virtually unused/unknown dict.viewXXX()
methods that are the exact? equivalent to the dict.XXX() methods in Python
3.

> By "separate list" I mean a snapshot of the dict at the time, and by
> "integrated iter()" I mean changes to the dict during iteration are
> seen by the iter.
>
> In Python 3 the iter_* methods replaced the list() type methods, which
> makes sense from the point-of-view of moving to a more iterator based
> language; however, as a result of that change the typical "iterate over
> a dict" operation now has a built-in gotcha: modifying the dict during
> the iteration can now cause exceptions.
>
> The solution, of course, is simple: surround the iterator call with
> list():
>
> list(dict.keys())
> list(dict.items())
> list(dict.values())
>
> for k, v in list(flag._value2member_map_.items()):
> ...
>
> The solution, however, feels a lot more boilerplate-ish. Either the
> programmer takes a lot more care to remember the current state of the dict
> (since it's no longer a snapshot), or "list()" is sprinkled around every
> iterator access.
>
> In other words, what used to be a completely safe operation now is not.
>
> Thoughts?

Is code that was written for Python 3 riddled with list(...) calls?

That's not my impression.

Do you see

RuntimeError: dictionary changed size during iteration

regularly?

I don't, and while I wouldn't have changed the dict interface I think Python
3's items() and values() -- who uses keys()? -- are clearly the better
defaults.

[Ethan Furman]

On 01/07/2017 03:04 PM, Peter Otten wrote:
> Ethan Furman wrote:
>
>> In Python 2 we have:
>>
>> dict().keys() \
>> dict().items() --> separate list() of the results
>> dict().values() /
>>
>> and
>>
>> dict().iter_keys() \
>> dict().iter_items() --> integrated iter() of the results
>> dict().iter_values() /
>
> I guess you didn't use these as often as I did ;)
>
> By the way, there are also the virtually unused/unknown dict.viewXXX()
> methods that are the exact? equivalent to the dict.XXX() methods in Python
> 3.

Ah, right -- in fact, those are the ones that got transplanted, I think.

>> In other words, what used to be a completely safe operation now is not.
>>
>> Thoughts?
>
> Is code that was written for Python 3 riddled with list(...) calls?

Generally, or just around dicts? I know I've placed list() around a few dict
calls (when I am going to modify the dict in the loop) or I make a separate list
to save the modifications and the bulk change in a separate loop.

> Do you see
>
> RuntimeError: dictionary changed size during iteration
>
> regularly?

No, thank goodness. Still embarrassing when it happens, though. :(

Thinking on it some more, I'm pretty sure I would have ended up with another
exception of a different flavor, so overall no extra harm done. ;)

--
~Ethan~

[Steve D'Aprano]

On Sun, 8 Jan 2017 08:48 am, Ethan Furman wrote:

> In Python 2 we have:
>
> dict().keys() \
> dict().items() --> separate list() of the results
> dict().values() /
>
> and
>
> dict().iter_keys() \
> dict().iter_items() --> integrated iter() of the results
> dict().iter_values() /

You missed another group of methods:

viewkeys()
viewitems()
viewvalues()

which are integrated, iterable, set-like views of the keys/items/values.

> By "separate list" I mean a snapshot of the dict at the time, and by
> "integrated iter()" I mean changes to the dict during iteration are
> seen by the iter.
>
> In Python 3 the iter_* methods replaced the list() type methods,

Its actually the view* methods.

> which
> makes sense from the point-of-view of moving to a more iterator based
> language; however, as a result of that change the typical "iterate over
> a dict" operation now has a built-in gotcha: modifying the dict during
> the iteration can now cause exceptions.

Even in Python 2, modifying the dict during iteration can cause exceptions:

for key in adict: # iterate over the dict directly
...


As usual, the standard rule applies: don't add or remove elements of a
collection while you are iterating over it.

This doesn't work either:

for i, x in enumerate(alist):
if condition(x):
del alist(i)


and has a similar solution:

for i, x in enumerate(list(alist)):
...

except that's more commonly written using slicing:

for i, x in enumerate(alist[:]):
...


For both dicts and lists, it is safe to modify existing elements:

adict[key] = new_value # provided key already exists

alist[index] = new_value


are both safe, but insertions and deletions are not.

> The solution, of course, is simple: surround the iterator call with
> list():

There's nothing magical about list. You could use tuple, or (sometimes) set
or frozenset. But list is good.

> list(dict.keys())
> list(dict.items())
> list(dict.values())
>
> for k, v in list(flag._value2member_map_.items()):
> ...
>
> The solution, however, feels a lot more boilerplate-ish. Either the
> programmer takes a lot more care to remember the current state of the dict
> (since it's no longer a snapshot), or "list()" is sprinkled around every
> iterator access.

Not *every* view access. Only the ones where you insert/delete elements.

> In other words, what used to be a completely safe operation now is not.
>
> Thoughts?

The old Python 2 keys/values/items methods used to make a copy of the
elements, whether you needed a copy or not. Now making a copy is your
responsibility. That feels more Pythonic to me.



--
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.