New to Go - map/filter higher-order functions on collections?

[Sean Talts]

Hi everyone,

I'm wondering if there are plans or it would be useful to implement map and filter on top of native Go collections, or maps and slices at least. Is there a reason that was not included? I've read a bit about why generics has not been included on a language level and instead they have been special cased on collections, but I don't see that precluding type-safe (i.e. not just translating everything to interface{} and back) implementations of map and filter. On the contrary, it seems to my admittedly Go-newbie self that they would, in fact, solve most of the use cases for generics. Thoughts? Does this violate some fundamental Go design decision I haven't yet encountered?

Thanks in advance,

Sean

PS I tried searching for this on the mailing list but my search-fu is weak and terms like "map" and "filter" are not particularly googleable. If I've missed a thread, feel free to link it! Thanks again.

[Ian Lance Taylor]

On Tue, Feb 4, 2014 at 5:28 PM, Sean Talts <xit...@gmail.com> wrote:
>
> I'm wondering if there are plans or it would be useful to implement map and
> filter on top of native Go collections, or maps and slices at least. Is
> there a reason that was not included? I've read a bit about why generics has
> not been included on a language level and instead they have been special
> cased on collections, but I don't see that precluding type-safe (i.e. not
> just translating everything to interface{} and back) implementations of map
> and filter. On the contrary, it seems to my admittedly Go-newbie self that
> they would, in fact, solve most of the use cases for generics. Thoughts?
> Does this violate some fundamental Go design decision I haven't yet
> encountered?

I'm reluctant to add increasingly special purpose builtin functions.
There is no obvious endpoint as we head down that road.

Ian

[John Souvestre]

Hello Ian.

> I'm reluctant to add increasingly special purpose builtin functions.

I believe that many would consider map, filter and reduce to be fundamental (not
special purpose) functional programming functions.

> There is no obvious endpoint as we head down that road.

By taking one step at a time, then reevaluating, you can avoid overshooting and
achieve a good balance. Not doing anything because the final solution isn't
apparent only guarantees that you won't get there. :)

Map, filter and reduce would be a good step, in my newbie opinion.

John

John Souvestre - New Orleans LA - (504) 454-0899

--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Ian Lance Taylor]

On Tue, Feb 4, 2014 at 8:59 PM, John Souvestre <jo...@sstar.com> wrote:
>
> > I'm reluctant to add increasingly special purpose builtin functions.
>
> I believe that many would consider map, filter and reduce to be fundamental (not
> special purpose) functional programming functions.
>
> > There is no obvious endpoint as we head down that road.
>
> By taking one step at a time, then reevaluating, you can avoid overshooting and
> achieve a good balance. Not doing anything because the final solution isn't
> apparent only guarantees that you won't get there. :)
>
> Map, filter and reduce would be a good step, in my newbie opinion.

Thinking about it a bit more, I'm not sure it makes sense even if we
ignore the slippery slope. The copy and append functions provide
facilites that are difficult to write in Go. The proposed map,
filter, and reduce functions presumably would call a function on each
element of a slice. That function would be provided by the caller,
and would have to have types that match the slice. But that means
that any use of these functions can be written as a trivial loop.

That is, copy and append get a significant advantage in working on
slices of any type. Map, filter and reduce do not get that advantage,
because they already require a function whose type must match the type
of the slice.

Ian

[John Souvestre]

Hello Ian.

If you want to accomplish the same functionality (as map, filter, reduce -
M/F/R) in another way, fine. My concern would be does it take more or less
code, and is the result easier or harder to read?

Even a simple loop takes 3 lines and allows more opportunity for errors. Also,
a M/F/R is often easier to make concurrent-safe.

> Map, filter and reduce do not get that advantage, because they
> already require a function whose type must match the type of
> the slice.

M/F/R do things that copy and append don't, so I don't follow your comparison.

John

John Souvestre - New Orleans LA - (504) 454-0899


-----Original Message-----
From: golan...@googlegroups.com [mailto:golan...@googlegroups.com] On
Behalf Of Ian Lance Taylor
Sent: Tue, February 04, 2014 11:04 pm
To: John Souvestre
Cc: golang-nuts
Subject: Re: [go-nuts] New to Go - map/filter higher-order functions on
collections?

[andrey mirtchovski]

> Even a simple loop takes 3 lines and allows more opportunity for errors.

the modern programmer's mentality appears to be that the best program
is the one not written.

[Caleb Spare]

On Tue, Feb 4, 2014 at 9:19 PM, John Souvestre <jo...@sstar.com> wrote:

Hello Ian.

If you want to accomplish the same functionality (as map, filter, reduce -
M/F/R) in another way, fine.

That "other way" is how idiomatic Go code is written, today.

 

My concern would be does it take more or less
code, and is the result easier or harder to read?

Even a simple loop takes 3 lines and allows more opportunity for errors.

for i, e := range foo {

  bar[i] = f(e)

}

bar = map(foo, func(e T) { return f(e) })

Those are actually almost exactly the same number of characters. The map is arguably harder to read, with the nested braces and parens. The only real downside to the for loop is if you believe that newlines are somehow an inferior character, or that the number of them is the most important code metric.

Also, the map would be much, much slower than that for loop without some very clever compiler optimizations.  

 

Also,
a M/F/R is often easier to make concurrent-safe.

Please provide evidence for this claim.

 

 > Map, filter and reduce do not get that advantage, because they
 > already require a function whose type must match the type of
 > the slice.

M/F/R do things that copy and append don't, so I don't follow your comparison.

append gets advantages as a builtin (avoiding having to specify types) which don't really apply to map/filter/reduce because you end up writing the types in your anonymous function anyway.

[John Souvestre]

"Everything should be as simple as possible, but not simpler."
--- Albert Einstein

John

John Souvestre - New Orleans LA - (504) 454-0899


-----Original Message-----
From: golan...@googlegroups.com [mailto:golan...@googlegroups.com] On
Behalf Of andrey mirtchovski
Sent: Tue, February 04, 2014 11:28 pm
To: John Souvestre
Cc: Ian Lance Taylor; golang-nuts
Subject: Re: [go-nuts] New to Go - map/filter higher-order functions on
collections?

[chris dollin]

On 5 February 2014 05:29, Caleb Spare <ces...@gmail.com> wrote:

bar = map(foo, func(e T) { return f(e) })

Surely

    bar = map(foo, f)

?  The function literal is superfluous. Yes?

Chris

--
Chris "allusive" Dollin

[Carlos Castillo]

The f(e) represents arbitrary code, not an actual function.

i.e. you could replace f(e) with:

•  e*e
• math.Sin(e)+math.Cos(e*2)
• a constant
• a 12 step process
• etc...
  

[Rob Pike]

The modern programmer thinks a newline is a thousand times harder to
type than any other character. If instead you take a newline as only
one keystroke, which it is, the fact that your program might not fit
on one line is a bearable burden.

-rob

[Carlos Castillo]

On Tuesday, February 4, 2014 8:59:18 PM UTC-8, johns wrote:

Hello Ian.

 > I'm reluctant to add increasingly special purpose builtin functions.

I believe that many would consider map, filter and reduce to be fundamental (not
special purpose) functional programming functions.

Go is not a functional programming language. Functional programming languages don't have for loops, and so need to provide assistance to perform iterative tasks.

 

 > There is no obvious endpoint as we head down that road.

By taking one step at a time, then reevaluating, you can avoid overshooting and
achieve a good balance.  Not doing anything because the final solution isn't
apparent only guarantees that you won't get there.  :)

Map, filter and reduce would be a good step, in my newbie opinion.

Newbies should learn the language they're trying to learn.

[John Souvestre]

Hello Rob.

>> bar = map(foo, f)

Certainly! What makes the map case simpler isn't that there are fewer lines,
but that there is less to think about - no "i" or "e" variables. Fewer lines is
just a bonus. :)

John

John Souvestre - New Orleans LA - (504) 454-0899


-----Original Message-----
From: golan...@googlegroups.com [mailto:golan...@googlegroups.com] On
Behalf Of Rob Pike
Sent: Wed, February 05, 2014 12:16 am
To: golan...@googlegroups.com
Subject: Re: [go-nuts] New to Go - map/filter higher-order functions on
collections?

[Carlos Castillo]

You seem to equate the function f with a single character, which it's not. The user must write it's implementation, which you have omitted for your own benefit.

[John Souvestre]

Hello Carlos.

 

I do not suggest that Go is a functional programming language nor would I hope that it ever would be.  This doesn’t mean that a few functional features would be a bad thing.

 

Couldn’t a “for … range” be considered at least semi-functional?  And Go isn’t an OO language either, but…  J

 

John

    John Souvestre - New Orleans LA - (504) 454-0899

 

[Jesse McNelis]

On Wed, Feb 5, 2014 at 4:29 PM, Caleb Spare <ces...@gmail.com> wrote:

bar = map(foo, func(e T) { return f(e) })

You mean,

bar := map(foo, func(e T)(T){return f(e)})

This would of course be rather useless since you'd most likely not want to copy and reallocate the entire slice. 

So more likely,

bar := map(foo, func(foo []T, i int, e T)(T){return f(e)})

You'd need two versions of map, one that mutated and one that copied.

You probably also want to return a slice of a different type for the times you want to map between types.

In this case map() would have to infer it's return type from the return type of the function passed to it.

This sounds really simple. I'm sure it will be great for the newbies.

--
=====================
http://jessta.id.au

[John Souvestre]

P.S.

 

Ø  Newbies should learn the language they're trying to learn.

 

I’m trying!  I believe I read that Go was an evolving language.  If so, I don’t see how the argument that “Go doesn’t do it now” is relevant.

 

On the other hand:  Simplicity, clarity, expressiveness, concurrency, and maintainability are all virtues which I believe Go tries to achieve.  I realize that language complexity (adding M/F/R would make Go more complex) is to be shunned.  So there has to be a balance.

 

If I’m out of line, I apologize. 

 

John

    John Souvestre - New Orleans LA - (504) 454-0899

 

From: golan...@googlegroups.com [mailto:golan...@googlegroups.com] On Behalf Of Carlos Castillo

Sent: Wed, February 05, 2014 12:24 am
To: golan...@googlegroups.com

[John Souvestre]

Hello Carlos.

 

f() has to be implemented regardless of whether you use a loop or a map.

 

John

    John Souvestre - New Orleans LA - (504) 454-0899

 

--

[Carlos Castillo]

Functional programming languages have features which make sense for how the user is expected to program in them:

• Loops don't exist (most func languages make it hard to have/use variables which would be needed to exit loops properly)
• Functions can be expressed very simply because they are often expected to be a single expression (eg: lambdas)

In go, we have loops, functions can span multiple line's / pages, and have a syntax that allow that body of code of that size to be legible. Adding these functions saves little to no typing on the user's part, and adds functionality which already exists, for a problem which has already been solved.

for ... range, allows you to succinctly express an iteration over the elements of a slice, map, or channel. Functional languages don't have loop control structures, so I don't see how it's "semi-functional".

[Bakul Shah]

The power of the map/filter abstraction starts becoming more apparent when you do things like

Filter(someFunc, Map(funcWith3args, vecA, vecB, vecC))

And you realize you should swap map and filter for your particular app for better performance.

In the example above, just for the map, you have to check that the three vectors are of same length, allocate a new vector, deal with error conditions, etc. So it is more than just a simple loop over range.

And it would be somewhat easier for a compiler to parallelize map/filter than infer a loop is parallelizable. And how you should slice up array computation for best performance would be machine/memory arch. dependent. I would want to do this in well optimized runtime code rather than have that sort of code littered through my program and which may be suboptimal for another machine.

Further extensions to channels & hash tables are obvious!

Sure, you can do all this without generic filters and maps but the essential logic and similarity gets lost in loops and error checking and what not.

A "generic" go to standard go source filter (gofront?) seems doable at first blush. Surprised that no one seems to have tried this experiment (or may be I just don't know about it). It would then be much easier to see if it has any merit.

[chris dollin]

On 5 February 2014 06:13, Carlos Castillo <cook...@gmail.com> wrote:

The f(e) represents arbitrary code, not an actual function.

SIlly of me to miss that.

[egon]

On Wednesday, February 5, 2014 10:31:30 AM UTC+2, Bakul Shah wrote:

The power of the map/filter abstraction starts becoming more apparent when you do things like

Filter(someFunc, Map(funcWith3args, vecA, vecB, vecC))

And you realize you should swap map and filter for your particular app for better performance.

Filter(someFunc, Map(funcWith3args, A, B, C))

vs. 

tmp := make([]Thing, len(A))

for i, a := range A {

tmp[i] = someFunc(a, B[i], C)

}

res := make([]Thing, 0, len(tmp))

for v := range tmp {

if someFilter(v) {

res = append(res, v)

}

}

Using range makes it obvious that there is a performance hit in the first place.

+ egon

[Carlos Castillo]

Yes, but if it's defined outside the map operation it's somewhat unfair to not count it towards the complexity of map but ignore that such a function is equally usable in a for loop:

func f (x int) int { ... }

list = map(list, f) // in-place map

for i, x := range list { 

      list[i] = f(x) 

}

Although the map case is shorter, it is not neither drastically so, nor is it significantly safer.

[egon]

On Wednesday, February 5, 2014 10:31:30 AM UTC+2, Bakul Shah wrote:

The power of the map/filter abstraction starts becoming more apparent when you do things like

Filter(someFunc, Map(funcWith3args, vecA, vecB, vecC))

And you realize you should swap map and filter for your particular app for better performance.

In the example above, just for the map, you have to check that the three vectors are of same length, allocate a new vector, deal with error conditions, etc. So it is more than just a simple loop over range.

And it would be somewhat easier for a compiler to parallelize map/filter than infer a loop is parallelizable. And how you should slice up array computation for best performance would be machine/memory arch. dependent. I would want to do this in well optimized runtime code rather than have that sort of code littered through my program and which may be suboptimal for another machine.

If you really want to make a case for a feature you have to show several real world problems (not a demonstration, not an example... a real world example that you would see running on some system) and show how the syntax helps and  do a proper comparison of using/not using it... And not just "it's two characters shorter" or "it hides the complexity".

The best performance is also OS/algorithm specific, so the compiler can't take care of it.

+ egon

[Bakul Shah]

On Feb 5, 2014, at 1:03 AM, egon <egon...@gmail.com> wrote:

If you really want to make a case for a feature you have to show several real world problems (not a demonstration, not an example... a real world example that you would see running on some system) and show how the syntax helps and  do a proper comparison of using/not using it... And not just "it's two characters shorter" or "it hides the complexity".

I agree 100%!

The best performance is also OS/algorithm specific, so the compiler can't take care of it.

Can be parameterized. 

[Gerard]

Yet another bikeshed colour discussion. And of course, Go's Godwin, the generics, are added to the topic.

Why do people think they have the miracle solution for non-existing hypothetical problems? I don't get it. Go != C++ != any other (functional) language

Can we please move on?

Gerard

[egon]

On Wednesday, February 5, 2014 11:10:43 AM UTC+2, Bakul Shah wrote:

On Feb 5, 2014, at 1:03 AM, egon <egon...@gmail.com> wrote:

If you really want to make a case for a feature you have to show several real world problems (not a demonstration, not an example... a real world example that you would see running on some system) and show how the syntax helps and  do a proper comparison of using/not using it... And not just "it's two characters shorter" or "it hides the complexity".

I agree 100%!

The best performance is also OS/algorithm specific, so the compiler can't take care of it.

Can be parameterized. 

Yup... 100 parameters would sound about right... then it will be definitely possible to find the best optimization settings that takes care of everything.

:/

+ egon

[Bienlein]

Hi Sean,

what you would like to have cannot be done in a type safe way without parameterized types. You will end up having to convert in your closure handed in to filter, map, etc. from interface{} to the actual type. That make using types pretty useless in the end. Gen is some attempt to add generics to Go: http://clipperhouse.github.io/gen/ I think this might come closest to what you would like to have.

As what generics are concerned I'm not sure whether they will ever arrive for Go. My take at Go is that it is some modernized C with special emphasis on concurrency (channels/select) and multi-threading (goroutines). People that define the function to obtain the size of an array as Len(myArray) instead of myArray.Len() really don't care about the world beyond C like generics and stuff.

Cheers, Bienlein

[John Souvestre]

Hello Carlos.

 

Ø  Yes, but if it's defined outside the map operation it's somewhat unfair to not count it towards the complexity of map but ignore that such a function is equally usable in a for loop:

 

Certainly the function has to be counted in both cases.  I don’t believe I said otherwise.

 

Ø  Although the map case is shorter, it is not neither drastically so, nor is it significantly safer.

 

In this example I would consider the main advantage to be the elimination of the 2 variables (i, x).

 

John

    John Souvestre - New Orleans LA - (504) 454-0899

 

From: golan...@googlegroups.com [mailto:golan...@googlegroups.com] On Behalf Of Carlos Castillo
Sent: Wed, February 05, 2014 2:54 am
To: golan...@googlegroups.com
Subject: Re: [go-nuts] New to Go - map/filter higher-order functions on collections?

 

Yes, but if it's defined outside the map operation it's somewhat unfair to not count it towards the complexity of map but ignore that such a function is equally usable in a for loop:

[Konstantin Khomoutov]

On Wed, 5 Feb 2014 01:43:43 -0800 (PST)
Bienlein <jet...@web.de> wrote:

[...]

> People that define the function to obtain the size of
> an array as Len(myArray) instead of myArray.Len() really don't care
> about the world beyond C like generics and stuff.

People who want to get the length of an array using myArray.Len() have
their brains indoctrinated by their first (or serious) programming
language learned, which is Java or C++ or C# these days.

I'm not implying such people are necessarily wrong; I just felt the
formulation of your statement somehow painted X.Len() as being in some
hidden sense superiour to len(X) which, in my opinion, requires proofs.

[Ian Lance Taylor]

On Tue, Feb 4, 2014 at 10:45 PM, John Souvestre <jo...@sstar.com> wrote:
>
> I’m trying! I believe I read that Go was an evolving language. If so, I
> don’t see how the argument that “Go doesn’t do it now” is relevant.

To be honest, I'm not sure I would describe Go as an evolving language
at this point. The language will continue to change, and indeed every
new release to date has brought in minor language enhancements (though
none have been proposed for the upcoming 1.3 release). But we've said
that we believe that the language is ready for use in production, and
we've made the Go 1 guarantee (http://golang.org/doc/go1compat). I
think that at this time it's more important to grow the Go ecosystem
and to improve the various implementations than it is to modify the
language proper.

This is not meant to say that people should stop proposing language
enhancements. Obviously Go is not perfect. But we aren't going to
make changes casually.

> On the other hand: Simplicity, clarity, expressiveness, concurrency, and
> maintainability are all virtues which I believe Go tries to achieve. I
> realize that language complexity (adding M/F/R would make Go more complex)
> is to be shunned. So there has to be a balance.

Yes, exactly: there has to be a balance. It's not enough to say that
new builtin functions would be useful--as Rob likes to say, of course
they would be useful, or you wouldn't have proposed them. They have
to be useful enough to outweigh the disadvantages of adding them to
the language.

In this case, as I said before, in my opinion, these functions are
only minor enhancements over what can already be written in Go. Yes,
they are enhancements. But they are minor. And I do think the
slippery slope argument has some validity here: I don't think we
should let the lack of generics in Go lead us to adding generics one
function at a time.

> If I’m out of line, I apologize.

You are not out of line.

Ian

[Jesper Louis Andersen]

On Wed, Feb 5, 2014 at 3:35 PM, Ian Lance Taylor <ia...@golang.org> wrote:

Yes, exactly: there has to be a balance.  It's not enough to say that
new builtin functions would be useful--as Rob likes to say, of course
they would be useful, or you wouldn't have proposed them.  They have
to be useful enough to outweigh the disadvantages of adding them to
the language.

One good way of keeping a language small is to mandate each proposal for a new feature also proposes the removal of another one. This way, the size of the language is kept and a new proposal has to subsume some other way of doing things. Otherwise, it won't work.

--
J.

[Henrik Johansson]

It is something to say for the virtue of not just adding stuff to a language.

http://joearms.github.io/2014/02/01/big-changes-to-erlang.html

Apparently Erlang just got maps!
They had records sure but still...

Going slow can be frustrating but it is not without its hard earned merits.

--

[Sean Talts]

Whoa, this thread really exploded while I slept! Let me first state that I am, in fact, trying to learn Go and avoid shoehorning my preconceptions into it. I consider this email exploration to be part of that :)

 

The primary benefit of map and filter (to me) is not in reducing typing, or eliminating new lines. It's entirely an issue of readability. 

 

For example, a common issue that is brought up on this mailing list (and which appears in our codebase) is translating an array of object type to an array of an interface type which that object implements. I understand why this translation is necessary, and it is trivial to write using the for loop pattern:

 

  func (p Product) Inventory() []InventoryResponse {

    r := make([]integrations.InventoryResponse, len(p.inventory))

    for index, inv := range p.inventory {

      r[index] = inv

    }

    return r

  }

  

However, the for loop pattern is used everywhere in go. In order to check that the effect of this code is in fact to copy our list of Products into a list of InventoryResponses we have to closely read the entire function and check that everything matches properly. Many for loops are simply performing a map operation, and that is fine, but when a map does something more complex it can be a subtle change, and easily missed while reading the code. In other words, this code is bad at expressing the intent of the programmer.

 

If you use map to express this code, then it is immediately obvious to the reader that you are going to perform a 1-1 translation from the source array to a destination array. All that is left to the reader is to examine the enclosed function to understand what that translation is.

 

  func (p Product) Inventory() []InventoryResponse {

    return map(p.inventory, func(i ProductInventory) InventoryResponse {

        return i

    })

  }

 

I find this to be a major enhancement in readability, which outweighs the cost of having an additional method in the standard.

Thanks,

Sean

PS If name-spacing is an issue, they could exist in a collections library that would have to be imported? Not sure if that was a major concern behind new builtins. 

--
You received this message because you are subscribed to a topic in the Google Groups "golang-nuts" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-nuts/RKymTuSCHS0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-nuts...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the Google Groups "golang-nuts" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-nuts/RKymTuSCHS0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-nuts...@googlegroups.com.

[egon]

On Wednesday, February 5, 2014 6:51:38 PM UTC+2, Sean Talts wrote:

Whoa, this thread really exploded while I slept! Let me first state that I am, in fact, trying to learn Go and avoid shoehorning my preconceptions into it. I consider this email exploration to be part of that :)

General tip, learn the language and write at least 10,000 lines before proposing changes to it :).

 

 

The primary benefit of map and filter (to me) is not in reducing typing, or eliminating new lines. It's entirely an issue of readability. 

 

For example, a common issue that is brought up on this mailing list (and which appears in our codebase) is translating an array of object type to an array of an interface type which that object implements. I understand why this translation is necessary, and it is trivial to write using the for loop pattern:

  

 

  func (p Product) Inventory() []InventoryResponse {

    r := make([]integrations.InventoryResponse, len(p.inventory))

    for index, inv := range p.inventory {

      r[index] = inv

    }

    return r

  }

  

This translation isn't free... by writing it as a for-range loop you make it obvious it's performance overhead. Also, this would be more idiomatic:

func (p Product) Inventory() []Response {

  rs := make([]Response, 0, len(p.inventory))

  for _, v := range p.inventory {

    rs = append(rs, somefn(v))

  }

  return rs

}

However, the for loop pattern is used everywhere in go. In order to check that the effect of this code is in fact to copy our list of Products into a list of InventoryResponses we have to closely read the entire function and check that everything matches properly. Many for loops are simply performing a map operation, and that is fine, but when a map does something more complex it can be a subtle change, and easily missed while reading the code. In other words, this code is bad at expressing the intent of the programmer.

But it's good at expressing what the program does. Show examples where it does something more complex and it's impossible to write it better. I'm guessing you are trying to write one liners instead of separating things out to functions.

 

 

If you use map to express this code, then it is immediately obvious to the reader that you are going to perform a 1-1 translation from the source array to a destination array. All that is left to the reader is to examine the enclosed function to understand what that translation is.

 

  func (p Product) Inventory() []InventoryResponse {

    return map(p.inventory, func(i ProductInventory) InventoryResponse {

        return i

    })

  }

 

I find this to be a major enhancement in readability, which outweighs the cost of having an additional method in the standard.

Show a proper real world example... you are still showing snippets of the code.

[Bienlein]

Am Donnerstag, 6. Februar 2014 08:48:48 UTC+1 schrieb egon:

 

But it's good at expressing what the program does. Show examples where it does something more complex and it's impossible to write it better. I'm guessing you are trying to write one liners instead of separating things out to functions.

 

That kind of reasoning will probably end up in a situation where it is argued that there is no use for closures...

[Egon Elbre]

I didn't disagree with his initial statement. It's about tradeoffs... you lose some in intent and gain information about behavior.

My point is that, if you are writing bad code then the bad code should be fixed first... and then you can start talking about adding things that improve it.

I use maps/filters/reducers etc.... if they provide significant improvement.... and it's a trivial thing to add for a particular type since there aren't too many places that need it.

+ egon

[ceving]

Am Mittwoch, 5. Februar 2014 07:16:08 UTC+1 schrieb Rob 'Commander' Pike:

The modern programmer thinks a newline is a thousand times harder to
type than any other character. If instead you take a newline as only
one keystroke, which it is, the fact that your program might not fit
on one line is a bearable burden.

I would say the Go designers are frightened of newlines. In particular the one in front of the hanging brace.

[Artem Titoulenko]

On Thursday, February 6, 2014 2:48:48 AM UTC-5, egon wrote:

On Wednesday, February 5, 2014 6:51:38 PM UTC+2, Sean Talts wrote:

Whoa, this thread really exploded while I slept! Let me first state that I am, in fact, trying to learn Go and avoid shoehorning my preconceptions into it. I consider this email exploration to be part of that :)

General tip, learn the language and write at least 10,000 lines before proposing changes to it :).

How are you counting the lines? Are these all whole lines that I wrote or do edited lines count as well? If I have really long lines, could those count as two lines or should I split them up to increase my score further?

 

 

The primary benefit of map and filter (to me) is not in reducing typing, or eliminating new lines. It's entirely an issue of readability. 

 

For example, a common issue that is brought up on this mailing list (and which appears in our codebase) is translating an array of object type to an array of an interface type which that object implements. I understand why this translation is necessary, and it is trivial to write using the for loop pattern:

  

 

  func (p Product) Inventory() []InventoryResponse {

    r := make([]integrations.InventoryResponse, len(p.inventory))

    for index, inv := range p.inventory {

      r[index] = inv

    }

    return r

  }

  

This translation isn't free... by writing it as a for-range loop you make it obvious it's performance overhead. Also, this would be more idiomatic:

I would like to believe that most devs know how map works and understand that it applies a function to all elements. It should also be obvious that there is a performance overhead involved with that. I'm not convinced that the lack of the `for` keyword makes engineers forget basic principles.

I would argue that it could be more idiomatic to use maps since that is a stronger signal of intent (apply function to all elements, return array of results) than a generic for/range which may be used for absolutely anything. Also consider the benefits of the compiler knowing about the intent of a loop and how it could optimize based on that information. Map can very easily be parallelized and split among all available cores evenly. This reduces the amount of guess-work that the parser/compiler has to do about what a loop is supposed to do and may speed up both compile and run time.

 

func (p Product) Inventory() []Response {

  rs := make([]Response, 0, len(p.inventory))

  for _, v := range p.inventory {

    rs = append(rs, somefn(v))

  }

  return rs

}

However, the for loop pattern is used everywhere in go. In order to check that the effect of this code is in fact to copy our list of Products into a list of InventoryResponses we have to closely read the entire function and check that everything matches properly. Many for loops are simply performing a map operation, and that is fine, but when a map does something more complex it can be a subtle change, and easily missed while reading the code. In other words, this code is bad at expressing the intent of the programmer.

But it's good at expressing what the program does. Show examples where it does something more complex and it's impossible to write it better. I'm guessing you are trying to write one liners instead of separating things out to functions.

Ideally functions should be concise and perform a single task. A more complex function could bring out issues that can be far more dangerous than anything a map implementation can do. Go has the ability to be an expressive language, where programmers can reap the benefits of using a compiled, static-typed language while spending less time on boiler plate and book-keeping. Having to rewrite the same basic functions for every method goes against that ideal and results in a lot of repeated, boilerplate, code; the very thing thing that Go fights against. Applying a function to elements of a slice is a basic task that shouldn't be reimplemented over and over again for every type.

 

 

 

If you use map to express this code, then it is immediately obvious to the reader that you are going to perform a 1-1 translation from the source array to a destination array. All that is left to the reader is to examine the enclosed function to understand what that translation is.

 

  func (p Product) Inventory() []InventoryResponse {

    return map(p.inventory, func(i ProductInventory) InventoryResponse {

        return i

    })

  }

 

I find this to be a major enhancement in readability, which outweighs the cost of having an additional method in the standard.

Show a proper real world example... you are still showing snippets of the code.

This is a real world example. Would 10,000 lines be more likely to convince you?

As a general idea, having first-class functions without higher order functions is a missed opportunity.

[Ian Lance Taylor]

On Thu, Feb 6, 2014 at 12:31 PM, Artem Titoulenko
<artem.ti...@gmail.com> wrote:
>
> Applying a function to elements
> of a slice is a basic task that shouldn't be reimplemented over and over
> again for every type.

I don't mean to be flip, but: http://golang.org/doc/faq#generics .

Ian

[Sean Talts]

Ian, did you mean to imply that you guys simply haven't found a good way to implement map and filter, as the content at the link suggests? Or perhaps that you'd rather find a good general purpose solution to generics instead (potentially because you don't want to pollute the default namespace, or because adding functions at the compiler level is a pain in the ass)?

I'm noticing as I look around the gc source that adding append to the language was quite complicated, so I have a better feeling for why you guys are reluctant to add more generic functions like this. That said, I still think it would be nice to have a more full-fledged collections library (or equivalent) at our disposal.

I'm going to spend some time this weekend trying to add map and filter to the language, so if it was the first case up above I'd be happy to submit a pull request. Otherwise we can just use my fork in the office, hah.

Thanks,

Sean

PS Are there any docs available about the go compiler? I haven't been able to find any after a few minutes googling... Any links would be appreciated! 

[Ian Lance Taylor]

On Thu, Feb 6, 2014 at 2:28 PM, Sean Talts <sean....@gmail.com> wrote:
>
> Ian, did you mean to imply that you guys simply haven't found a good way to
> implement map and filter, as the content at the link suggests? Or perhaps
> that you'd rather find a good general purpose solution to generics instead
> (potentially because you don't want to pollute the default namespace, or
> because adding functions at the compiler level is a pain in the ass)?

I was replying specifically to the comment I quoted: "Applying a

function to elements of a slice is a basic task that shouldn't be

reimplemented over and over again for every type." That is the
argument for generics. I don't think we can take that argument and
make it an argument for map and filter. If we do, we really are on a
slippery slope, one that ends with us adding every function that can
be described as a "basic task" to the language. The argument for
adding map and filter to the language has to be better than that.

> I'm noticing as I look around the gc source that adding append to the
> language was quite complicated, so I have a better feeling for why you guys
> are reluctant to add more generic functions like this.

The implementation complexity is far less important than the language
complexity.

Ian

[Sean Talts]

On Thu, Feb 6, 2014 at 7:42 PM, Ian Lance Taylor <ia...@golang.org> wrote:

On Thu, Feb 6, 2014 at 2:28 PM, Sean Talts <sean....@gmail.com> wrote:
>
> Ian, did you mean to imply that you guys simply haven't found a good way to
> implement map and filter, as the content at the link suggests? Or perhaps
> that you'd rather find a good general purpose solution to generics instead
> (potentially because you don't want to pollute the default namespace, or
> because adding functions at the compiler level is a pain in the ass)?

I was replying specifically to the comment I quoted: "Applying a

function to elements of a slice is a basic task that shouldn't be

reimplemented over and over again for every type."  That is the
argument for generics.  I don't think we can take that argument and
make it an argument for map and filter.  If we do, we really are on a
slippery slope, one that ends with us adding every function that can
be described as a "basic task" to the language.  The argument for
adding map and filter to the language has to be better than that.

Is that not the same argument for append and copy? Is it just a question of where to draw the line, or are there other criteria involved?

Thanks,

Sean

[Ian Lance Taylor]

On Thu, Feb 6, 2014 at 4:59 PM, Sean Talts <sean....@gmail.com> wrote:
>
>
> On Thu, Feb 6, 2014 at 7:42 PM, Ian Lance Taylor <ia...@golang.org> wrote:
>>
>> On Thu, Feb 6, 2014 at 2:28 PM, Sean Talts <sean....@gmail.com> wrote:
>> >
>> > Ian, did you mean to imply that you guys simply haven't found a good way
>> > to
>> > implement map and filter, as the content at the link suggests? Or
>> > perhaps
>> > that you'd rather find a good general purpose solution to generics
>> > instead
>> > (potentially because you don't want to pollute the default namespace, or
>> > because adding functions at the compiler level is a pain in the ass)?
>>
>> I was replying specifically to the comment I quoted: "Applying a
>> function to elements of a slice is a basic task that shouldn't be
>> reimplemented over and over again for every type." That is the
>> argument for generics. I don't think we can take that argument and
>> make it an argument for map and filter. If we do, we really are on a
>> slippery slope, one that ends with us adding every function that can
>> be described as a "basic task" to the language. The argument for
>> adding map and filter to the language has to be better than that.
>
>
>
> Is that not the same argument for append and copy? Is it just a question of
> where to draw the line, or are there other criteria involved?

The append and copy builtins can not easily be written in Go, not with
the same efficiency. That is not true of map and filter.

Ian

[Kyle Lemons]

On Wed, Feb 5, 2014 at 8:51 AM, Sean Talts <sean....@gmail.com> wrote:

Whoa, this thread really exploded while I slept! Let me first state that I am, in fact, trying to learn Go and avoid shoehorning my preconceptions into it. I consider this email exploration to be part of that :)

 

The primary benefit of map and filter (to me) is not in reducing typing, or eliminating new lines. It's entirely an issue of readability. 

 

For example, a common issue that is brought up on this mailing list (and which appears in our codebase) is translating an array of object type to an array of an interface type which that object implements.

It might be commonly brought up, but I don't think it should happen very often in real-world code.  In my experience, if you have to make this translation it means you're fighting the language somehow.

 

I understand why this translation is necessary, and it is trivial to write using the for loop pattern:

 

  func (p Product) Inventory() []InventoryResponse {

    r := make([]integrations.InventoryResponse, len(p.inventory))

    for index, inv := range p.inventory {

      r[index] = inv

    }

    return r

  }

  

However, the for loop pattern is used everywhere in go. In order to check that the effect of this code is in fact to copy our list of Products into a list of InventoryResponses we have to closely read the entire function and check that everything matches properly. Many for loops are simply performing a map operation, and that is fine, but when a map does something more complex it can be a subtle change, and easily missed while reading the code. In other words, this code is bad at expressing the intent of the programmer.

 

If you use map to express this code, then it is immediately obvious to the reader that you are going to perform a 1-1 translation from the source array to a destination array. All that is left to the reader is to examine the enclosed function to understand what that translation is.

 

  func (p Product) Inventory() []InventoryResponse {

    return map(p.inventory, func(i ProductInventory) InventoryResponse {

        return i

    })

  }

 

I find this to be a major enhancement in readability, which outweighs the cost of having an additional method in the standard.

I actually find this to be a rather poor example... "return i" makes it look like it's a no-op conversion.  If I saw this, I would have to scratch my head and hunt around for why it wasn't simply a copy().  On the other hand, this seems clearer:

ir := make([]InventoryResponse, len(products))

for i, p := range products {

  ir[i] = p

}

It's still not obvious that there's a type conversion, but the source variable and destination type are much less buried in boilerplate.

[egon]

On Thursday, February 6, 2014 10:31:51 PM UTC+2, internet mage wrote:

On Thursday, February 6, 2014 2:48:48 AM UTC-5, egon wrote:

On Wednesday, February 5, 2014 6:51:38 PM UTC+2, Sean Talts wrote:

Whoa, this thread really exploded while I slept! Let me first state that I am, in fact, trying to learn Go and avoid shoehorning my preconceptions into it. I consider this email exploration to be part of that :)

General tip, learn the language and write at least 10,000 lines before proposing changes to it :).

How are you counting the lines? Are these all whole lines that I wrote or do edited lines count as well? If I have really long lines, could those count as two lines or should I split them up to increase my score further?

The suggestion is more of a metaphor rather than a hard requirement. It refers to Gladwells 10,000 hours of practice rule... If you don't know how to properly use a hammer you shouldn't try to randomly make changes to it...

 

 

The primary benefit of map and filter (to me) is not in reducing typing, or eliminating new lines. It's entirely an issue of readability. 

 

For example, a common issue that is brought up on this mailing list (and which appears in our codebase) is translating an array of object type to an array of an interface type which that object implements. I understand why this translation is necessary, and it is trivial to write using the for loop pattern:

  

 

  func (p Product) Inventory() []InventoryResponse {

    r := make([]integrations.InventoryResponse, len(p.inventory))

    for index, inv := range p.inventory {

      r[index] = inv

    }

    return r

  }

  

This translation isn't free... by writing it as a for-range loop you make it obvious it's performance overhead. Also, this would be more idiomatic:

I would like to believe that most devs know how map works and understand that it applies a function to all elements. It should also be obvious that there is a performance overhead involved with that. I'm not convinced that the lack of the `for` keyword makes engineers forget basic principles. 

I would argue that it could be more idiomatic to use maps since that is a stronger signal of intent (apply function to all elements, return array of results) than a generic for/range which may be used for absolutely anything.

You are still not analyzing the feature properly... if all you can come up with what are the pros of a feature then you haven't examine the feature well enough.

 

Also consider the benefits of the compiler knowing about the intent of a loop and how it could optimize based on that information.

 

Map can very easily be parallelized and split among all available cores evenly.

No they can not... What about cache misses? What if several things run on the same machine? How do you know when it's appropriate to do that if you have only a abstract definition of the function?

 

This reduces the amount of guess-work that the parser/compiler has to do about what a loop is supposed to do and may speed up both compile and run time. 

 

func (p Product) Inventory() []Response {

  rs := make([]Response, 0, len(p.inventory))

  for _, v := range p.inventory {

    rs = append(rs, somefn(v))

  }

  return rs

}

However, the for loop pattern is used everywhere in go. In order to check that the effect of this code is in fact to copy our list of Products into a list of InventoryResponses we have to closely read the entire function and check that everything matches properly. Many for loops are simply performing a map operation, and that is fine, but when a map does something more complex it can be a subtle change, and easily missed while reading the code. In other words, this code is bad at expressing the intent of the programmer.

But it's good at expressing what the program does. Show examples where it does something more complex and it's impossible to write it better. I'm guessing you are trying to write one liners instead of separating things out to functions.

Ideally functions should be concise and perform a single task. A more complex function could bring out issues that can be far more dangerous than anything a map implementation can do. Go has the ability to be an expressive language, where programmers can reap the benefits of using a compiled, static-typed language while spending less time on boiler plate and book-keeping. Having to rewrite the same basic functions for every method goes against that ideal and results in a lot of repeated, boilerplate, code; the very thing thing that Go fights against. Applying a function to elements of a slice is a basic task that shouldn't be reimplemented over and over again for every type.

 

 

 

If you use map to express this code, then it is immediately obvious to the reader that you are going to perform a 1-1 translation from the source array to a destination array. All that is left to the reader is to examine the enclosed function to understand what that translation is.

 

  func (p Product) Inventory() []InventoryResponse {

    return map(p.inventory, func(i ProductInventory) InventoryResponse {

        return i

    })

  }

 

I find this to be a major enhancement in readability, which outweighs the cost of having an additional method in the standard.

Show a proper real world example... you are still showing snippets of the code.

This is a real world example. Would 10,000 lines be more likely to convince you?

func (p Product) Inventory() []Response {

rs := make([]Response, 0, len(p.inventory))

for _, v := range p.inventory {

rs = append(rs, v)

}

return rs

}

Here the code takes 2 characters more... or alternatively:

type items []Inventory

func (items items) map(fn func(v Inventory)) []Response {

rs := make([]Response, 0, len(items))

for _, v := range p.inventory {

rs = append(rs, fn(v))

}

return rs

}

func (p Product) Inventory() []Response {

return items(p.inventory).map(func(i Inventory) Response {

return i

})

}

The amount of code doesn't matter... you haven't shown the problem you are having. If that is the problem that you have, that a single function that be written by a map, then it's trivial to fix... either provide a map function for it or just use a for loop. Assuming your editor has some form of templates, you can write a template for writing a map function...

If you are doing a lot of those conversions then the problem could be somewhere else in your code (poor model, poor code, poor structuring).... that's why you need to provide the full code... so the code can be analyzed properly.

+ egon

[egon]

Also it's useful to also add the requirements, otherwise the analysis could be way off...

[egon]

On Friday, February 7, 2014 12:28:26 AM UTC+2, Sean Talts wrote:

Ian, did you mean to imply that you guys simply haven't found a good way to implement map and filter, as the content at the link suggests? Or perhaps that you'd rather find a good general purpose solution to generics instead (potentially because you don't want to pollute the default namespace, or because adding functions at the compiler level is a pain in the ass)?

I'm noticing as I look around the gc source that adding append to the language was quite complicated, so I have a better feeling for why you guys are reluctant to add more generic functions like this. That said, I still think it would be nice to have a more full-fledged collections library (or equivalent) at our disposal.

I'm going to spend some time this weekend trying to add map and filter to the language, so if it was the first case up above I'd be happy to submit a pull request. Otherwise we can just use my fork in the office, hah.

I suggest starting with a design document... e.g. https://code.google.com/p/go-wiki/wiki/DesignDocuments ... you need to understand what the downsides of your solution are and whether at all it will fix your problem. I.e. how will it affect readability; how easily can other people interop with your code, how likely are other people to use your code, what happens if the map results don't fit in memory, how will you maintain your fork, what are all the downsides, what are all the upsides, are there examples that can show how it improves the code significantly. Also, a source-generator for map is trivial, modifying the compiler is not...

I believe you are trying to solve the wrong problem and with creating a fork you are also creating more work for yourself. It's a fun thing to do, but it won't solve your readability problem... btw. if you are using a lot of mappings/transformations then using FBP style code could help.

+ egon