Is it necessary to change the behavior of maps.Keys and maps.Values?

[fliter]

Since the x/exp/maps era, I have used maps.Keys and maps.Values ​​extensively. They are very useful and avoid users from doing loop iterations.

I remember that they were added to the standard library, but I used maps.Keys in the latest go version and was surprised that it could not compile. By looking at the source code, I found that their behavior seemed to have changed for iter?

Related discussions:

https://github.com/golang/go/issues/57436

https://github.com/golang/go/issues/61538

Maybe I have limited information and less knowledge than the members of the go team, but if it is to support the functions of the iter package, why not add two new function names, such as maps.KeysIter and maps.ValuesIter, instead of changing the behavior of the previous method?

In addition, how do you deal with the scenarios previously handled by maps.Keys? Do you do it manually with for ..range..append, or use golang.org/x/exp/maps? Is it necessary to propose a proposal to add these two methods back to the standard library, but with different names?

[Sebastien Binet]

Hi,

If you want a slice in lieu of an iterator, you can use the slices package and its Collect function :

https://pkg.go.dev/slices#Collect

hth,
-a

Dec 24, 2024 05:08:01 fliter <imc...@gmail.com>:

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/golang-nuts/4f95be15-1a7d-4cb8-a19d-eb33a1736d68n%40googlegroups.com.

[Jason Phillips]

Both maps.Keys and maps.Values exist in the standard library, https://pkg.go.dev/maps#Keys

Perhaps I'm missing some nuance about the difference between x/exp/maps and the standard library implementation?

[Def Ceb]

x/exp/maps.Keys() returned a slice, while stdlib maps.Keys() returns an
iterator. Can't just switch between them transparently in almost all cases.
Sebastian was simply saying that slices.Collect() could be used as a
stop-gap solution for moving from x/exp/maps to stdlib maps, since every
current occurrence of maps.Keys(foo) could be converted to
slices.Collect(maps.Keys(foo)) to restore previous behaviour. Then you
can check that it still compiles and the tests still pass. Then maybe
consider reworking the code to use iterators instead of slices altogether.

Anyway, x/exp is explicitly experimental and not to be relied upon. This
interface change could just as easily have happened within x/exp itself
in a version bump rather than during its move to the standard library.
If iterators had been added after maps was moved to the standard
library, then the iterator variant definitely would have appeared as
maps.KeysIter(), but they have no promises to keep with x/exp code. And
since converting an iterator to a slice is as simple as wrapping the
iterator in slices.Collect() if a slice is really needed, it's not a big
issue.

Jason Phillips:

> Both maps.Keys and maps.Values exist in the standard library, https://
> pkg.go.dev/maps#Keys
>
> Perhaps I'm missing some nuance about the difference between x/exp/maps
> and the standard library implementation?
>
> On Tuesday, December 24, 2024 at 2:31:04 AM UTC-5 Sebastien Binet wrote:
>
> Hi,
>
> If you want a slice in lieu of an iterator, you can use the slices
> package and its Collect function :
>

> https://pkg.go.dev/slices#Collect <https://pkg.go.dev/slices#Collect>

>
> hth,
> -a
>
> Dec 24, 2024 05:08:01 fliter <imc...@gmail.com>:
>
>
>
> Since the x/exp/maps era, I have used maps.Keys and
> maps.Values ​​extensively. They are very useful and avoid users
> from doing loop iterations.
>
> I remember that they were added to the standard library, but I
> used maps.Keys in the latest go version and was surprised that
> it could not compile. By looking at the source code, I found
> that their behavior seemed to have changed for iter?
>
> Related discussions:
>
> https://github.com/golang/go/issues/57436 <https://github.com/

> golang/go/issues/57436>
>
> https://github.com/golang/go/issues/61538 <https://github.com/
> golang/go/issues/61538>

>
> Maybe I have limited information and less knowledge than the
> members of the go team, but if it is to support the functions of
> the iter package, why not add two new function names, such as
> maps.KeysIter and maps.ValuesIter, instead of changing the
> behavior of the previous method?
>
> In addition, how do you deal with the scenarios previously
> handled by maps.Keys? Do you do it manually with

> for ..range..append, or use golang.org/x/exp/maps <http://
> golang.org/x/exp/maps>? Is it necessary to propose a proposal to

> add these two methods back to the standard library, but with
> different names?
>
> --
> You received this message because you are subscribed to the
> Google Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from
> it, send an email to golang-nuts...@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/
> golang-nuts/4f95be15-1a7d-4cb8-a19d-

> eb33a1736d68n%40googlegroups.com <https://groups.google.com/d/
> msgid/golang-nuts/4f95be15-1a7d-4cb8-a19d-
> eb33a1736d68n%40googlegroups.com?
> utm_medium=email&utm_source=footer>.

>
> --
> You received this message because you are subscribed to the Google
> Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send

> an email to golang-nuts...@googlegroups.com <mailto:golang-
> nuts+uns...@googlegroups.com>.

> To view this discussion visit https://groups.google.com/d/msgid/golang-

> nuts/e249451f-21a7-4e61-9df2-875c712a92ccn%40googlegroups.com <https://
> groups.google.com/d/msgid/golang-nuts/
> e249451f-21a7-4e61-9df2-875c712a92ccn%40googlegroups.com?
> utm_medium=email&utm_source=footer>.

[Amnon]

There are big advantages in maps.Keys and maps.Values returning iterators.
It allows us to iterate very big maps without having to allocate vast amounts of memory.

When  maps was moved from x/exp to the standard library, the Keys and Values methods were not added
until 1.23.0 when iterators became available. This way it was not necessary to add  new method names.

The Go team are perfectly at liberty to change the API of packages in x/exp, especially when these packages migrate to the standard 
library. This is why x/exp exists in the first place, to allow experimentation, in a way which does not lock mistakes in to all future releases.

And yes, it would be possible to add KeysAsSlice and ValuesAsSlice methods to the stdlib. But this would increase the API surface
of the package for no real benefit.

[Mike Schinkel]

On Dec 28, 2024, at 2:41 PM, Amnon <amn...@gmail.com> wrote:

There are big advantages in maps.Keys and maps.Values returning iterators.
It allows us to iterate very big maps without having to allocate vast amounts of memory.

I definitely agree there is a big advantage to have functions in the standard library that return iterators for maps. 

What is a shame, however, is that those functions did not get a naming convention to indicate them to be different from those that just return a slice of keys or a slice of values much like how the suffix `Func` is used as a naming convention to indicate that its parameters differ.  As is they squat on obvious names for functions that just return slices.

It would have been less confusing to learn and remember had they named them maps.KeysIterator and maps.ValuesIterator, maps.KeysSeq and maps.ValuesSeq, maps.KeysRange and maps.ValuesRange, or with some other suffix. But sadly, given the backward compatibility guarantee of Go, that ship has sailed.

-Mike

[Axel Wagner]

I don't see a good argument for the lack of a suffix "naturally" implying returning a slice. I don't think `ValuesSlice` would be less natural than `ValuesSeq`.

At the end of the day, it's purely a question of what you think would be more frequently used. That should get the better name. `Values` (as is) composes naturally with `slices.Collect` and so you get both in one function and it seems a good choice for the default option. There *is* an advantage of a single `ValuesSlice` function over the composed version, because it can pre-allocate. But that seems a rather special advantage that can live with a special name. And either way, I'd argue that we could/should probably get rid of that advantage anyways, by inlining both `Values` *and* `Collect` and optimize the resulting `append` loop.

In any case, the existence of this discussion is a historical accident. We added `x/exp/maps.Values` before iterators existed, as an experiment, so had to use slices. We *intentionally* waited with adding `maps.Values` until *after* iterators got added, to be able to give it the better name. If we would have never had `x/exp/maps.Values` or if we had added `range` over func before generics (and AFAIK it has been proposed before generics), we would have only ever had a `maps.Values` that returned iterators. No one would have even noticed.

--

You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

To view this discussion visit https://groups.google.com/d/msgid/golang-nuts/8064C03C-51B9-4046-A5C2-88380A978FEE%40newclarity.net.

[Amnon]

A nice thing about Go is that it is easy on the eye. Names are short, and easy to read. When you look at Go code, you are not faced with a 
wall of black text. The signal to noise ratio is high. Variable and function names general convey what they do, what they mean, rather than
the types of their return values. So we would tend to call a variable `count` rather than `countInteger`.

I don't have any pets. But if I had a dog, I might have named it fido. As a Go programmer, I would not have named it fidoDog.
In the Go world, we do try to avoid verbosity, and long convoluted names.

Another thing which makes Go easy to learn and easy to understand is that the API surface is generally small. 
I am glad that the Go team are doing their best to keep it that way. 

[Mike Schinkel]

> On Dec 29, 2024, at 1:28 AM, Axel Wagner <axel.wa...@googlemail.com> wrote:
>
> At the end of the day, it's purely a question of what you think would be more frequently used. That should get the better name. `Values` (as is) composes naturally with `slices.Collect` and so you get both in one function and it seems a good choice for the default option.

In my projects I use the sequence versions less frequently so having maps.Keys() and maps.Values() return an actual slice would be preferable. As is, I am forced to use the far less better name of `maps.Collect(maps.Keys())` instead of just `maps.Keys()`.

Why don't I use the sequence versions more frequently? Because I try not to create huge in-memory maps and instead prefer to move logic that manipulates large amounts of data into a proper database.

But when I do it would use large maps it would make sense to call that out with maps.KeysSeq() et. al.

> I don't see a good argument for the lack of a suffix "naturally" implying returning a slice. I don't think `ValuesSlice` would be less natural than `ValuesSeq`.
>

> In any case, the existence of this discussion is a historical accident. We added `x/exp/maps.Values` before iterators existed, as an experiment, so had to use slices. We *intentionally* waited with adding `maps.Values` until *after* iterators got added, to be able to give it the better name. If we would have never had `x/exp/maps.Values` or if we had added `range` over func before generics (and AFAIK it has been proposed before generics), we would have only ever had a `maps.Values` that returned iterators. No one would have even noticed.

Ironically you alluded to the good argument you did not see, in your third paragraph.

The good argument is precedence and consistency. I do agree that had it not been for the historical "accident" things would be different, but history is not something you can merely hand-wave away; it exists and has significant influence even when there are those who would prefer to ignore it.

Had that history never occurred I would agree that a lack of a suffix would not naturally imply a slice. But that history did occur, and naming is not good when it ignores the context that occurred before it.

The decision to ignore precedence and consistency is taking steps toward the "fractal of bad design" that is PHP and its horrendous inconsistency in standard library function naming and parameter usage that people have complained about for decades.

Anyway, I am not sure why you felt the need to add your voice to defend it because, as I wrote, "That ship has sailed."

-Mike

[Mike Schinkel]

> On Dec 29, 2024, at 1:38 AM, Amnon <amn...@gmail.com> wrote:
>
> A nice thing about Go is that it is easy on the eye. Names are short, and easy to read. When you look at Go code, you are not faced with a wall of black text. The signal to noise ratio is high. Variable and function names general convey what they do, what they mean, rather than the types of their return values. So we would tend to call a variable `count` rather than `countInteger`.

> <snip>

> I am glad that the Go team are doing their best to keep it that way.

Then you must REALLY hate that the Go team chose these names in the standard library:

# strings Package
- Builder.Write() vs Builder.WriteString()
- Builder.WriteByte() vs Builder.WriteRune()
- Reader.Read() vs Reader.ReadString()
- Reader.ReadByte() vs Reader.ReadRune()

# bytes Package
- Buffer.Write() vs Buffer.WriteString()
- Buffer.WriteByte() vs Buffer.WriteRune()
- Reader.Read() vs Reader.ReadString()
- Reader.ReadByte() vs Reader.ReadRune()

# strconv Package
- strconv.AppendBool()
- strconv.AppendFloat()
- strconv.AppendInt()
- strconv.AppendUint()
- strconv.FormatBool()
- strconv.FormatFloat()
- strconv.FormatInt()
- strconv.FormatUint()
- strconv.ParseBool()
- strconv.ParseFloat()
- strconv.ParseInt()
- strconv.ParseUint()

# slices Package
- slices.Sort() vs slices.SortFunc()
- slices.IsSorted() vs slices.IsSortedFunc()
- slices.BinarySearch() vs slices.BinarySearchFunc()
- slices.Contains() vs slices.ContainsFunc()
- slices.Index() vs slices.IndexFunc()
- slices.LastIndex() vs slices.LastIndexFunc()

# maps Package
- maps.Equal() vs maps.EqualFunc()

# bufio Package
- Reader.Read() vs Reader.ReadString()
- Reader.ReadByte() vs Reader.ReadRune()
- Writer.Write() vs Writer.WriteString()
- Writer.WriteByte() vs Writer.WriteRune()

# archive/tar Package
- Reader.Read() vs Reader.ReadString()
- Writer.Write() vs Writer.WriteString()

-Mike

[Axel Wagner]

Why don't I use the sequence versions more frequently? Because I try not to create huge in-memory maps and instead prefer to move logic that manipulates large amounts of data into a proper database. 

Iterators are more efficient for small maps as well. Allocating and then throwing away small slices increases GC pressure. I got easily 10% or more performance improvements by refactoring a loop over a slice with a loop over an iterator.

Then you must REALLY hate that the Go team chose these names in the standard library:

# strings Package
- Builder.Write() vs Builder.WriteString()

- Builder.WriteByte() vs Builder.WriteRune()
- Reader.Read() vs Reader.ReadString()
- Reader.ReadByte() vs Reader.ReadRune()

# bytes Package
- Buffer.Write() vs Buffer.WriteString()
- Buffer.WriteByte() vs Buffer.WriteRune()
- Reader.Read() vs Reader.ReadString()
- Reader.ReadByte() vs Reader.ReadRune()

All of these exist to implement different interface with different tradeoffs. For example, `WriteString` implements `io.StringWriter`, which is more efficient than calling `Write(string(p))`, because the latter would make the argument escape, thanks to the virtual call.

Yes, it would be better to have simpler names. But given the nature of interfaces in Go, we need two different methods to do two different things and they need to have two different names. At least one of them, unfortunately, has to be bad.

That doesn't disprove the principle, though.

 

# strconv Package
- strconv.AppendBool()
- strconv.AppendFloat()
- strconv.AppendInt()
- strconv.AppendUint()
- strconv.FormatBool()
- strconv.FormatFloat()
- strconv.FormatInt()
- strconv.FormatUint()
- strconv.ParseBool()
- strconv.ParseFloat()
- strconv.ParseInt()
- strconv.ParseUint()

 

# slices Package
- slices.Sort() vs slices.SortFunc()
- slices.IsSorted() vs slices.IsSortedFunc()
- slices.BinarySearch() vs slices.BinarySearchFunc()
- slices.Contains() vs slices.ContainsFunc()
- slices.Index() vs slices.IndexFunc()
- slices.LastIndex() vs slices.LastIndexFunc()

# maps Package
- maps.Equal() vs maps.EqualFunc()

Indeed. If we had more powerful generics (and had them from the beginning) we could have chosen singular functions with simpler names here.

Perhaps, in the future, we *will* get those and can clean up here.

 

# bufio Package
- Reader.Read() vs Reader.ReadString()
- Reader.ReadByte() vs Reader.ReadRune()
- Writer.Write() vs Writer.WriteString()
- Writer.WriteByte() vs Writer.WriteRune()

# archive/tar Package
- Reader.Read() vs Reader.ReadString()
- Writer.Write() vs Writer.WriteString()

As above.

 

-Mike

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

To view this discussion visit https://groups.google.com/d/msgid/golang-nuts/28732884-0E56-4FAA-9DA4-C39700870CC7%40newclarity.net.

[Mike Schinkel]

On Dec 29, 2024, at 5:11 PM, Axel Wagner <axel.wa...@googlemail.com> wrote:

Why don't I use the sequence versions more frequently? Because I try not to create huge in-memory maps and instead prefer to move logic that manipulates large amounts of data into a proper database.  

Iterators are more efficient for small maps as well. Allocating and then throwing away small slices increases GC pressure. I got easily 10% or more performance improvements by refactoring a loop over a slice with a loop over an iterator.

By the same token, I do not create lots of throw-away slices, either, and especially not in logic that would generate hundreds or thousands of them.  

If you do that then no wonder you find improvements in GC pressure.

All of these exist to implement different interface with different tradeoffs. For example, `WriteString` implements `io.StringWriter`, which is more efficient than calling `Write(string(p))`, because the latter would make the argument escape, thanks to the virtual call.

Yes, it would be better to have simpler names. But given the nature of interfaces in Go, we need two different methods to do two different things and they need to have two different names. At least one of them, unfortunately, has to be bad.

That doesn't disprove the principle, though.

Depends on what principle(s) you are referring to.  

If you are referring to the principles of precedence and consistency, it actually does.

Indeed. If we had more powerful generics (and had them from the beginning) we could have chosen singular functions with simpler names here.  Perhaps, in the future, we *will* get those and can clean up here.

On that we can certainly agree.

-Mike

[tapi...@gmail.com]

Please note that "maps.Collect" is slow. https://github.com/golang/go/issues/68261

It is best to write your own custom Collect function.

[tapi...@gmail.com]

On Monday, December 30, 2024 at 6:12:10 AM UTC+8 Axel Wagner wrote:

Why don't I use the sequence versions more frequently? Because I try not to create huge in-memory maps and instead prefer to move logic that manipulates large amounts of data into a proper database. 

Iterators are more efficient for small maps as well. Allocating and then throwing away small slices increases GC pressure. I got easily 10% or more performance improvements by refactoring a loop over a slice with a loop over an iterator.

This is typical micro benchmark. There are too such "allocating and then throwing away small slices" cases in Go programming.

The iterator cases are just a tiny portion of them.

[tapi...@gmail.com]

On Tuesday, December 31, 2024 at 9:42:50 AM UTC+8 tapi...@gmail.com wrote:

On Monday, December 30, 2024 at 6:12:10 AM UTC+8 Axel Wagner wrote:

Why don't I use the sequence versions more frequently? Because I try not to create huge in-memory maps and instead prefer to move logic that manipulates large amounts of data into a proper database. 

Iterators are more efficient for small maps as well. Allocating and then throwing away small slices increases GC pressure. I got easily 10% or more performance improvements by refactoring a loop over a slice with a loop over an iterator.

This is typical micro benchmark. There are too such "allocating and then throwing away small slices" cases in Go programming.

The iterator cases are just a tiny portion of them.

And the iterator cases have alternative no-allocation ways. Many other cases have not.

[fliter]

Thank you all for the discussion, it was very helpful!

[Axel Wagner]

> Please note that "maps.Collect" is slow. https://github.com/golang/go/issues/68261

Please note that I did address that in my message.

CollectN matters most for non-trivial iterators, e.g. where you first transform an iterator using `xiter.Map` or the like.

Something like slices.Collect(maps.Keys(m)) - which this discussion is about - can be relatively easily optimized by the compiler, if we want.

> This is typical micro benchmark. There are too such "allocating and then throwing away small slices" cases in Go programming. The iterator cases are just a tiny portion of them.

I was talking about end-to-end execution time of the program. In a CPU-bound program, it is not uncommon for allocations in the inner loop to have significant effects.

[Xu Liu]

On Tue, Dec 31, 2024 at 1:40 PM Axel Wagner <axel.wa...@googlemail.com> wrote:

> Please note that "maps.Collect" is slow. https://github.com/golang/go/issues/68261

Please note that I did address that in my message.

CollectN matters most for non-trivial iterators, e.g. where you first transform an iterator using `xiter.Map` or the like.

Something like slices.Collect(maps.Keys(m)) - which this discussion is about - can be relatively easily optimized by the compiler, if we want.

I doubt it is easily.

 

> This is typical micro benchmark. There are too such "allocating and then throwing away small slices" cases in Go programming. The iterator cases are just a tiny portion of them.

I was talking about end-to-end execution time of the program. In a CPU-bound program, it is not uncommon for allocations in the inner loop to have significant effects.

Let's focus the specific topic. Do you ranging over map.Keys and map.Values is faster than directly range over maps?