retaining state in for comprehensions

[Rich Morin]

I recently posted an issue (#3114) about elixir_for.erl, suggesting that an option be added to for/1 to allow retention of variables between iterations.  See:

  add option to retain local variables

  https://github.com/elixir-lang/elixir/issues/3114

  https://github.com/elixir-lang/elixir/blob/master/lib/elixir/src/elixir_for.erl

  http://wiki.cfcl.com/Projects/Elixir/Repetition#Variable_Reversion

José Valim commented as follows:

Please bring this discussion to the core mailing list. Before we make this work, we actually should decide if we want to make this work. We had similar discussion in the past and people were more inclined to a "no" because it relies on using variables to pass state/accumulate instead of explicitly accumulating it (which would be the more functional approach).

I would like to it to be more explicit too but I have never found a good syntax for it.

There are already ways to retain state between iterations of "for" comprehensions (eg, other processes, the process dictionary).  I'm just suggesting that we make them prettier, more efficient, etc.

Elixir draws very few lines in the sand regarding how programmers should write their code. When it does so (eg, no mutable shared state), the reasons tend to be clear, objective, and pragmatic. Are there any such reasons in this case?

I'd rather not have the language making stylistic decisions for me. If I want that kind of "help", I know where to find it. Keeping state in loop variables may not always be the best idea, but I'd like to have the option.

I don't have strong feelings about the syntax. How about something like:

  iex> i = 21

  21

  iex> for n <- (1..2), retain_state do

  ...>   i = i + i

  ...>   IO.puts i

  ...> end

  42

  84

  [:ok, :ok]

-r

[Greg Vaughn]

So, now I need to remember the special case that when I'm in a list comprehension with the option of retain_state, then the = no longer means match as it does everywhere else in the language, but instead means assignment like in imperative languages?

No. I prefer Elixir avoid this.

-Greg Vaughn

> --
> You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/d001daf4-cdf3-4a7f-8225-cb3fb7bc9279%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Chris McCord]

I would prefer we continue to recommend a reduce operation. I feel like this kind of feature would be abused and lead towards an imperative style, and ultimately less ideal implementations.

Sent from my iPhone

> To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/B8127F7F-EA16-417A-B076-883E9051A4C5%40gmail.com.

[Patrick Gombert]

Could you provide an example where reduce wouldn't work? I'm just having trouble imagining a use case. I think functions in Enum might already solve this.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/B8127F7F-EA16-417A-B076-883E9051A4C5%40gmail.com.

[José Valim]

Thanks Rich for bringing the discussion here.

I'd rather not have the language making stylistic decisions for me. If I want that kind of "help", I know where to find it. Keeping state in loop variables may not always be the best idea, but I'd like to have the option.

Right, except this is not a stylistic decision. Nowhere in the language we use assignment as a state. Sure, we have variables, they keep values, but they are never treated as the state of the operation. So this is a conceptual issue and that's why I asked it to be discussed here.

Someone down below asked if reduce is the equivalent here, the answer is yes. In fact, for comprehensions are implemented on top of reduce.

[Rich Morin]

José-

Thanks for prompting and encouraging this discussion. I'm not expert
at Elixir, Erlang, or Elixir, so please excuse and correct any details
I may get wrong. I'll start with some of the early responses, to get
them out of the way.

Greg Vaughn said:
> So, now I need to remember the special case that when I'm in a list
> comprehension with the option of retain_state, then the = no longer
> means match as it does everywhere else in the language, but instead
> means assignment like in imperative languages?

The = performs matching, possibly binding a variable to a value. The
retain_state flag (or whatever) would not change this. It would only
affect whether variable bindings would persist between iterations.

Chris McCord said:
> I feel like this kind of feature would be abused and lead towards
> an imperative style, and ultimately less ideal implementations.

Many of Elixir's features are ripe for abuse. For example, macros
are infamous as a pitfall for the unwary. I think it comes down to
a risk/reward trade-off.

"UNIX was not designed to stop its users from doing stupid things,
as that would also stop them from doing clever things." — Doug Gwyn

Patrick Gompert said:
> Could you provide an example where reduce wouldn't work? I'm just
> having trouble imagining a use case. I think functions in Enum
> might already solve this.

There are many ways to perform repetition while retaining state.
Indeed, as José says, "for comprehensions are implemented on top of
reduce". So, the real question, IMHO, is not whether Elixir should
allow repetition with retention, but what mechanisms it should have
and how convenient the mechanisms should be.

José Valim said:
> Right, except this is not a stylistic decision. Nowhere in the
> language we use assignment as a state. Sure, we have variables,
> they keep values, but they are never treated as the state of
> the operation. So this is a conceptual issue and that's why I
> asked it to be discussed here.

In many uses of recursion, a function saves some state in a data
structure, then passes the data structure to (another invocation
of) itself. What is the essential difference between passing a
map and retaining bindings on loop variables?


A use case might help to focus the discussion. Chris McCord's
book describes a "while" macro, which can be used as follows:

while ...condition... do
...something...
end

If a programmer needs to count the iterations, s/he might try:

count = 1
while ...condition... do
...something...
count = count + 1
end

The syntax would suggest (to most programmers) that this should work.
However, it does not (in Chris's version), because the macro is based
on a for comprehension.

So, the Principle of Least Astonishment seems top be violated. Worse,
there is no way (other than managing state retention explicitly via
a reduce or somesuch) for the programmer to get the desired effect.

-r

--
http://www.cfcl.com/rdm Rich Morin r...@cfcl.com
http://www.cfcl.com/rdm/resume San Bruno, CA, USA +1 650-873-7841

Software system design, development, and documentation

[Saša Jurić]

To give another twist to this discussion, a few months ago, while I played with Conway's Game of Life, I used a custom collectable to count the number of surrounding alive cells.

This is the comprehension: https://gist.github.com/sasa1977/6877c52c3c35c2c03c82#file-conway-ex-L82-L94

And here's the collectable: https://gist.github.com/sasa1977/6877c52c3c35c2c03c82#file-conway-ex-L4-L25

While this approach is no doubt an overkill in this case (I could have just collect into a list, and then sum it), I have a feeling the pattern might be useful in some cases where performance and/or memory usage matters, and creating an intermediate list is not a viable option.

I agree with others that Enum.reduce is more general, but I must also say that I like for comprehensions due to their elegant and rich support for multiple generators and filters, so I wonder whether supporting accumulator in a for comprehension can be somehow made simpler than what I did there. Don't have any suggestion though :/

[José Valim]

In many uses of recursion, a function saves some state in a data
structure, then passes the data structure to (another invocation
of) itself.  What is the essential difference between passing a
map and retaining bindings on loop variables?

The difference is that the state is being explicitly passed (as an argument) and returned (as the result of a function invocation) instead of being indirectly stored and refreshed inside a variable.

 

So, the Principle of Least Astonishment seems top be violated. 

 

Let's not bring the Principle of Least Astonishment into discussions. As Matz says, everyone has their own experience and the "principle" is going to trigger on different situations and different features. It is a subjective observation that does not apply to everyone.

 

Worse, there is no way (other than managing state retention explicitly via
a reduce or somesuch) for the programmer to get the desired effect.

For comprehensions are great for (flat)map+filtering, however, if you want to take, reduce, drop, reverse, one needs to get acquainted with the functions in Enum.

The biggest downside that the proposed syntax will only work for comprehensions. If someone tries to write:

    sum = 0

    Enum.each [1, 2, 3], fn x -> sum = sum + x end

    sum

It is not going to work and, opposite to for, it can't be made to work. We would be literally adding a special case to for comprehensions without any indication that it is a special case.

So I am going to be more incisive here: the proposed feature implementation that implicitly relies on variables to propagate state won't be accepted to the language. However, there is still a question out there, which is how can we bolt reduce into a for comprehension at least a bit more explicitly?

I am being more incisive because I believe it will be more productive for everyone if we focus on something that can eventually be part of the language.

Here is one example:

sum = 0

for x <- [1, 2, 3], with: sum do

  sum = sum + x

end

sum #=> 6

We use :with to tell the comprehension to pass an existing variable for every element. Although we are still using variables to pass state, we are at least being explicit which variables are being used.

There are very likely other approaches and ways to express this which we should explore.

[Ben Wilson]

It's examples like the following that I think show that we're just dealing with different notions of what kind of language elixir is:

count = 1 
  while ...condition... do 
    ...something... 
    count = count + 1 
  end 

This is imperative programming, we don't want this stuff. This is an enormous change from the language philosophy elixir has at present.

I think there's value to providing reduction abilities to the for comprehension, but the whole idea of mutating external state ought to be anathema. If you want to count your "while" loops, spin up an agent and send it messages.

[Greg Vaughn]

José I could get behind this example with one change. Remove the =. "sum = sum + x" now screams imperative programming to me thanks to Elixir. Also what about having the initial value of the 'into' var be inline?

for x <- [1, 2, 3], with: sum = 0, do: sum + x

-Greg Vaughn

[Josh Adams]

This is imperative programming, we don't want this stuff. This is an enormous change from the language philosophy elixir has at present.

I think there's value to providing reduction abilities to the for comprehension, but the whole idea of mutating external state ought to be anathema. If you want to count your "while" loops, spin up an agent and send it messages.

I was staying quiet mostly because I had lots of other stuff to do, but +1 on this comment.  If it will lead to me finding a lot of imperative elixir code in the wild, I want nothing to do with it.  I already suffer from writing too-imperative code in elixir *without* language features that would encourage it.  I have often had a design run afoul of something like the complained-about "lack of mutable state" in my code[1].  Inevitably, a little thought made it clear that my design was poor.  In ruby, I would have continued on with the poor design because nothing backhanded me the way that elixir's lexical scoping does when I made mistakes like this.  I love that Elixir stops me from making the design flaws that I usually find after running afoul of this.

Anyway, that's my $0.02

[1] I know SSA is not immutability, but this example comes far too close for comfort

[Josh Adams]

for x <- [1, 2, 3], with: sum = 0, do: sum + x

I'm going to channel the great Richard O'Keefe and suggest we need real examples here.  This is a trivial `reduce` and doesn't benefit from becoming a comprehension.  Do we have a real example of something that needs this?  Something other than a 1 or 2 line bit of frivolity?

-Josh 

[Patrick Gombert]

+1 Josh. I think it would be a good idea to avoid having two ways to use reduce without a good reason.

--

You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAA1-O0w-V0NDdR3y5xWVztZMQppP2zccXKQcS0NKciv%2BcMkruA%40mail.gmail.com.

[Sasa Juric]

To me, for is a great alternative to a nested reduce. Other than the Conway example I posted earlier, I don’t have anything specific at hand, but I occasionally write nested reduces (actually folds, since I’m doing it in Erlang) in prod, so I definitely see the need. I’ll watch out for such situations in future and try to come back with a concrete example.

Somewhat vaguely, I’d say an accumulator in a for can be useful if you need to reduce a Cartesian product of multiple enumerables, or a nested enumerable of enumerables, possibly with filtering.

--
You received this message because you are subscribed to a topic in the Google Groups "elixir-lang-core" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elixir-lang-core/_veg3CFQkS4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elixir-lang-co...@googlegroups.com.

[Peter Hamilton]

What if we created "reduce comprehensions"? I don't like the proposed " with: sum" because it doesn't really fit the traditional reduce function signature as it returns the list rather than the accumulator. Reduce comprehensions would be just like reduce but would support multiple generators and filters like for comprehensions.

That would give all the flexibility in the world plus a great deal of convenience.

Example:

reduce x <- [1,2,3], y <- [2,3,4], x + y == 5, acc: sum, do:
  sum + 1
end

We'd have to tinker with the account: part and provide an initial value, but I think it would be pretty useful.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAFOzVX_RFmF0K0aKMT6ZA8SbK9BBdTBJ1G3gVCG2TsJSAZsrSA%40mail.gmail.com.

[Ben Wilson]

I really like this actually. As far as the initial value goes, we could just supply it as something like

reduce x <- [1,2,3], y <- [2,3,4], x + y == 5, acc: 0, as: sum, do:
  sum + 1
end

It's a little verbose. I'm worried if we just do something like 

reduce x <- [1,2,3], y <- [2,3,4], x + y == 5, 0, acc: sum, do:
  sum + 1
end

that you have to look closely to figure out which of the items is the initial value given how many arguments it can take. Love the idea though.

[Peter Hamilton]

I guess my hopes are:

My worry is someone will go from comprehensions to Enum and ask questions like "how do I carry state from one interaction of map to another?" and come to conclusions like "reduce is like a for comprehension when you use the 'with' keyword" (that would definitely make me feel sad inside...)

An alternate syntax might be something like (credit: José):

for x <- [1,2,3], reduce:

acc, do

  acc +

x

end

Which is pretty similar to the 'with' keyword except we are calling it out under the correct technical name.

[Peter Hamilton]

Wow... Botched formatting from mobile... Apologies... Will correct and resend from laptop when I get off this train.

[Peter Hamilton]

The beauty of the current Stream and Enum implementations is the backbone of reduce and transform. I don't think we want to shelter the user from reduce and I don't think we should masquerade reduce as something else.

That's sort of what I see the discussion as. "Reduce is hard, so let's mess with other concepts that's are easier to grasp." By explicitly calling it a reduce there may be a learning curve, but reduce/fold/TCO are all non optional parts of a functional language.

I guess my hopes are:

1) don't overload terms (like for comprehensions) unnecessarily
2) Enum/Stream should mirror comprehensions whenever possible

My worry is someone will go from comprehensions to Enum and ask questions like "how do I carry state from one interaction of map to another?" and come to conclusions like "reduce is like a for comprehension when us you the 'with' keyword" (that would definitely make me feel sad inside...)

An alternate syntax might be something like (credit: José):

for x <- [1,2,3], reduce: acc, do
  acc + x
end

Which is pretty similar to the 'with' keyword except we are returning one value and calling it a reduce (the correct technical name).

[Saša Jurić]

I also like the distinct reduce comprehension. It seems clear and explicit, and as Peter said doesn’t pollute the mapping approach of the for comprehension.

You received this message because you are subscribed to a topic in the Google Groups "elixir-lang-core" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elixir-lang-core/_veg3CFQkS4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/9d70af37-e5a6-4ae6-bef8-ddc457d79b77%40googlegroups.com.

[Booker Bense]

My 2 cents. Comprehensions have always made me a bit uneasy. If you understand what's going on under the hood (i.e. it's just syntatic sugar around an Enum)

then that's fine. 

It allows you to express something in a way that is more familiar to people coming from imperative languages. 

However, that is a double edged sword in that you are bringing in all the intuition that people have build up around using for loops. 

Most of those intuitions are wrong/broken in a functional setting. I feel like comprehensions are largely a lie to the reader of the 

code about what you are actually doing. 

The reduce version of the comprehension seems the least objectionable, but I really feel like the whole concept is wrong in some way that

I can't easily state. To me this goes much deeper than any stylistic consideration. 

- Booker C. Bense 

[greggreg]

+1 on the distinct reduce comprehension, how about something like:

> c = :atom
:atom
#            ⇣iterator     ⇣iterator    ⇣guard     ⇣accumulator    
> r = reduce a <- [1,2,3], b <- [4,5,6], a + b < 8, c = 0 do
        c + a * b                
      end
45
> c
:atom
> r
45

The accumulator is always passed as the last item to  `reduce`.
The accumulator is implicitly set with each iteration.
Reduce returns the result of the reduction and no scope is polluted.

[Vincent Siliakus]

I think this would be confusing, as your example of the accumulator is currently also a valid guard in a comprehension. See https://github.com/zambal/eml/blob/master/lib/eml.ex#L477 for an example of this.

Op dinsdag 24 februari 2015 20:21:00 UTC+1 schreef greggreg:

[Vincent Siliakus]

I like Ben's suggestion most:

reduce n <- [1,2,3], acc: 0, as: sum do
  sum + n
end

It's explicit and (as a consequence) easy to read, at least IMHO. Adding this type of functionality as an option to for seems like a bad idea to me because, as others have pointed out, for stops being a comprehension that way.

-vincent

[Rich Morin]

Adding for-like syntax to reduce could resolve the issues I'm concerned
with, while retaining simplicity, clarity about what is being done, etc.
I like the idea of making variable retention explicit. This documents
intent in a very useful way, helping both human and mechanized analysis.

That said, I would like it to allow multiple variables as accumulators.
If I have (say) two conceptually independent counters, I shouldn't have
to pack and unpack them into an artificially-complected data structure.

[greggreg]

Yeah I definitely understand the concern, my counter is:

The reduce keyword would make it obvious that the last "guard" passed is actually the accumulator.
The accumulator would naturally be a simple statement, possibly only `identifier = expression`, that always matched and had no effect on the iterators. (I'm not sure if "iterator" is the correct term, someone chime in if otherwise)

I think this looks nice as well (`reduce into` seems easy to explain and copies `for`):

reduce a <- [1,2,3], b <- [4,5,6], a + b < 8, into: [c, 0] do


  c + a * b
end


or perhaps an implicit accumulator name (since there will always be one):

reduce a <- [1,2,3], b <- [4,5,6], a + b < 8, acc: 0  do
  @acc + a * b
end

I also like Ben's suggestion, it just seems a little clunky in it's verbosity to me.

[Vincent Siliakus]

Op dinsdag 24 februari 2015 22:37:59 UTC+1 schreef Rich Morin:

That said, I would like it to allow multiple variables as accumulators. If I have (say) two conceptually independent counters, I shouldn't have to pack and unpack them into an artificially-complected data structure.

I think it would be really hard to do without making it look like a statement/imperative construct.

[Ben Wilson]

Yeah I agree. I appreciate Rich's desire to make reduction easier, but I think there's some unfamiliarity with erlang and elixir's way of doing things. Functions don't return multiple values, if you want that use a tuple, and at the end of the day list comprehensions are function calls. It's for the same reason that we wouldn't want mutable state, functions can't mutate external state. {counter1, counter2} isn't "artificially" complicated.

[Vincent Siliakus]

Op dinsdag 24 februari 2015 22:43:35 UTC+1 schreef greggreg:

Yeah I definitely understand the concern, my counter is:

The reduce keyword would make it obvious that the last "guard" passed is actually the accumulator.
The accumulator would naturally be a simple statement, possibly only `identifier = expression`, that always matched and had no effect on the iterators. (I'm not sure if "iterator" is the correct term, someone chime in if otherwise)

My counter argument to this would be that it would make = look too much like assignment :)

Let me write down all reduce suggestions until now. To make it easier to compare, I let them all do the same operation:

Peter's suggestion
sum = 0
reduce n <- [1,2,3], acc: sum do
  sum + n
end

Ben's suggestions

reduce n <- [1,2,3], acc: 0, as: sum do
  sum + n
end

reduce n <- [1,2,3], 0, acc: sum do
  sum + n
end

Greggreg's suggestions
reduce n <- [1,2,3], sum = 0 do
  sum + n
end

reduce n <- [1,2,3], into: [sum, 0] do

  sum + n
end

reduce x <- [1,2,3], acc: 0 do
  @acc + x
end

The difference in verbosity is pretty minimal in my opinion. Now that I look at all suggestions together, maybe Peter's original suggestion is best. It's still explicit, but it doesn't have a 'magic' connection between acc: value and as: variable.

[Robert Virding]

Looking at this discussion as an "outsider" I do find parts for the discussion a little strange. Specifically that writing x = x + 1 feels very imperative. It does of course, but this was specifically allowed in elixir so as to have to NOT write the more declarative x1 = x0 + 1. Depending on which functional background you come from this could also be called a fold.

Robert

[Rich Morin]

On Feb 24, 2015, at 14:28, Ben Wilson <benwil...@gmail.com> wrote:
> Yeah I agree. I appreciate Rich's desire to make reduction easier, but
> I think there's some unfamiliarity with erlang and elixir's way of
> doing things. Functions don't return multiple values, if you want that
> use a tuple, and at the end of the day list comprehensions are function
> calls. It's for the same reason that we wouldn't want mutable state,
> functions can't mutate external state. {counter1, counter2} isn't
> "artificially" complicated.

Thanks for the clarification! To see if I'm on the right track, here
are some use cases that use a variation on Peter's suggestion:


reduce n <- [1,2,3], acc: { 0, 1 } do
{ sum, prod } = tuple
{ sum + n, prod * n }
end

This isn't too awful, but the code has to unpack the acc tuple's items
and the loop variables sum and prod. The reader also has to understand
the tuple's connascence of position.


reduce n <- [1,2,3], acc: { sum: 0, prod: 1 } do
{ sum: sum + n, prod: prod * n }
end

This trades connascence of position for connascence of name, which is
generally more comprehensible, robust, etc.

[Rich Morin]

Sorry, this use case has an oops:

reduce n <- [1,2,3], acc: { 0, 1 } do
{ sum, prod } = tuple
{ sum + n, prod * n }
end

This is probably more correct, but more verbose:

tuple = { 0, 1 }
reduce n <- [1,2,3], acc: tuple do

{ sum, prod } = tuple
{ sum + n, prod * n }
end

-r

[Peter Hamilton]

On Tue Feb 24 2015 at 2:39:46 PM Vincent Siliakus <zam...@gmail.com> wrote:

Op dinsdag 24 februari 2015 22:43:35 UTC+1 schreef greggreg:

Yeah I definitely understand the concern, my counter is:

The reduce keyword would make it obvious that the last "guard" passed is actually the accumulator.
The accumulator would naturally be a simple statement, possibly only `identifier = expression`, that always matched and had no effect on the iterators. (I'm not sure if "iterator" is the correct term, someone chime in if otherwise)

My counter argument to this would be that it would make = look too much like assignment :)

Let me write down all reduce suggestions until now. To make it easier to compare, I let them all do the same operation:

Peter's suggestion
sum = 0
reduce n <- [1,2,3], acc: sum do
  sum + n
end

I think I'm being misquoted. Let me correct :-)

I don't like assigning sum an initial value outside the comprehension. I think having the user assign sum and then implicitly assign sum each iteration is confusing. I also think that `acc: sum` might be read as `acc: 0` if `sum = 0` is right before it.

Let's take a different approach. In a for comprehension we introduce new variables with the <- operator (or the = operator, depending on whether it is a collection). On the left side is the new variable and the right side is an expression. Here, we want to introduce a new special variable and it's initial value.

I think that the following would be inline with the current behavior/syntax:

reduce n <- [1,2,3], m = 5 + 6, acc: sum = 0 do

  sum + n*m

end

It would result in 66. (0 + 11 + 22 + 33)

The question then becomes what can we allow in the `acc:` section. Surely something like:

reduce n <- [1,2,3], acc: sum = initial*5 do

  sum + n

end

would work fine.

Thoughts?

Ben's suggestions

reduce n <- [1,2,3], acc: 0, as: sum do
  sum + n
end

reduce n <- [1,2,3], 0, acc: sum do
  sum + n
end

Greggreg's suggestions
reduce n <- [1,2,3], sum = 0 do
  sum + n
end

reduce n <- [1,2,3], into: [sum, 0] do

  sum + n
end

reduce x <- [1,2,3], acc: 0 do
  @acc + x
end

The difference in verbosity is pretty minimal in my opinion. Now that I look at all suggestions together, maybe Peter's original suggestion is best. It's still explicit, but it doesn't have a 'magic' connection between acc: value and as: variable.

--

You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/de88b844-0207-4560-b25e-525736c15050%40googlegroups.com.

[Gilbert Kennen]

On 02/24/2015 01:43 PM, greggreg wrote:
>
> reduce a <- [1,2,3], b <- [4,5,6], a + b < 8, into: [c, 0] do
> c + a * b
> end
>
> or perhaps an implicit accumulator name (since there will always be one):
>
> reduce a <- [1,2,3], b <- [4,5,6], a + b < 8, acc: 0 do
> @acc + a * b
> end

Not to pick on you personally, but as the best example of a
comprehension offered so far.

I dislike cluttering up global namespace by making 'reduce' a reserved
term. It also doesn't seem to me that it is that much of an advantage to
do a reduce comprehension in a single step as opposed to two steps.

for a <- [1,2,3], b <- [4,5,6], a + b < 8, do: a * b
|> Enum.sum

or, to emulate a verbose example:

for a <- [1,2,3], b <- [4,5,6], a + b < 8, do: {a, b}
|> Enum.reduce(0, fn {a, b}, sum -> a * b + sum end)

I think the for comprehension is already complicated enough and either
adding on optional reduce functionality to it or making a reduce version
of it will only increase its complexity.

I could probably be on-board for a lazy comprehension which could reduce
memory usage with large collections, but I don't think we need anything
as drastic as reducing comprehensions.

[Ben Wilson]

I rather like this version, particularly if it would work to support full pattern matching, which would let us solve Rich's dual accumulator nicely ie:

reduce n <- [1,2,3], acc: {sum, prod} = {0,0} do

  {sum +n, sum * n}

end

[Ben Wilson]

Oops! I meant to actually use the prod variable for the multiplication

reduce n <- [1,2,3], acc: {sum, prod} = {0,0} do

  {sum + n, prod * n}

end

With the respect to the question of including another function in the default namespace, I think the unique position reduce holds in the functional world warrants its inclusion.

[Chris McCord]

reduce n <- [1,2,3], acc: {sum, prod} = {0,0} do

  {sum + n, prod * n}

end

Is this really a win over Enum reduce?

Enum.reduce [1,2,3], {0,0}, fn  n, {sum, prod} ->

  {sum + n, prod * n}

end

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/516e699f-c9f5-4481-be72-ab5c032547a7%40googlegroups.com.

[Rich Morin]

Minor nit repaired in examples: initial value for prod needs to be 1.


On Feb 24, 2015, at 19:00, Chris McCord <ch...@chrismccord.com> wrote:
>> reduce n <- [1,2,3], acc: {sum, prod} = {0,1} do

>> {sum + n, prod * n}
>> end
>
> Is this really a win over Enum reduce?
>

> Enum.reduce [1,2,3], {0,1}, fn n, {sum, prod} ->

> {sum + n, prod * n}
> end

I like to keep larger function definitions separate from their uses,
so I'd prolly write this more like:

iex> sum_prod = fn n, {sum, prod} -> { sum + n, prod * n } end
iex> Enum.reduce [1,2,3], {0,1}, sum_prod # => {6, 6}


In any case, the benefits become more apparent when there is more
than one operation that needs to be done for each iteration, eg:

reduce n <- [1,2,3], acc: {sum, prod} = {0,1} do
this_op
that_op
{ sum + n, prod * n }
end

[Peter Hamilton]

The win is when you have multiple collections and filters

reduce x <- [1,2,3, 4], y <- [3,4,5], x * y > 10, x * y < 14, acc: sum = 0 do
  sum + x + y
end

Compare that with

Stream.map([1,2,3,4], fn(x) ->
    Stream.map([3,4,5], &({x,&1})
  end) |> Stream.filter(fn({x,y}) ->
    x*y > 10 && x * y < 14
  end) |> Enum.reduce(0, fn ({x,y}, sum) -> sum + x + y end)

Granted, with a Stream.cross function it would look like:

Stream.cross([1,2,3,4],[3,4,5])
  |> Stream.filter(fn({x,y}) -> x*y > 10 && x * y < 14 end)
  |> Enum.reduce
  |> Enum.reduce(0, fn ({x,y}, sum) -> sum + x + y end)

Which isn't terrible. It still is a bit clumsy. Try adding a third dimension, z. In the reduce comprehension you add it to the comprehension and the block. In the Stream/Enum version you have to add it to every stage because now you are passing a 3-tuple instead of a two tuple.

The reduce comprehension syntax is fully macro-able:

iex(1)> quote do
...(1)>   reduce x <- [1,2,3], y <- [3,4,5], n = 100, acc: sum = 10 do
...(1)>     sum + x*y + n
...(1)>   end
...(1)> end
{:reduce, [],
 [{:<-, [], [{:x, [], Elixir}, [1, 2, 3]]},
  {:<-, [], [{:y, [], Elixir}, [3, 4, 5]]}, {:=, [], [{:n, [], Elixir}, 100]},
  [acc: {:=, [], [{:sum, [], Elixir}, 10]}],
  [do: {:+, [context: Elixir, import: Kernel],
    [{:+, [context: Elixir, import: Kernel],
      [{:sum, [], Elixir},
       {:*, [context: Elixir, import: Kernel],
        [{:x, [], Elixir}, {:y, [], Elixir}]}]}, {:n, [], Elixir}]}]]}
iex(2)>

 I would suggest that someone build it via a macro and use it for a bit before we try to get it into elixir-lang proper.

Any volunteers?

On Tue, Feb 24, 2015, 7:00 PM Chris McCord <ch...@chrismccord.com> wrote:

reduce n <- [1,2,3], acc: {sum, prod} = {0,0} do

  {sum + n, prod * n}

end

Is this really a win over Enum reduce?

Enum.reduce [1,2,3], {0,0}, fn  n, {sum, prod} ->

  {sum + n, prod * n}

end

On Feb 24, 2015, at 9:53 PM, Ben Wilson <benwil...@gmail.com> wrote:

Oops! I meant to actually use the prod variable for the multiplication

reduce n <- [1,2,3], acc: {sum, prod} = {0,0} do

  {sum + n, prod * n}

end

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/516e699f-c9f5-4481-be72-ab5c032547a7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/35A4877C-7300-49F7-8F44-5BF93B2D110A%40chrismccord.com.

[Ben Wilson]

I'll give it a shot tomorrow. For reference, where is the actual code for the for comprehension? The source link in the docs just takes you to https://github.com/elixir-lang/elixir/blob/74645bed76382cf015b3123d911aaa6534e6bf66/lib/elixir/lib/kernel/special_forms.ex#L1204

For that matter, where is any of the code for the special forms macros?

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/516e699f-c9f5-4481-be72-ab5c032547a7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.

[Rich Morin]

On Feb 24, 2015, at 20:04, Ben Wilson <benwil...@gmail.com> wrote:
> For reference, where is the actual code for the for comprehension?

https://github.com/elixir-lang/elixir/blob/master/lib/elixir/src/elixir_for.erl

[Saša Jurić]

+1 on what Peter said

To hopefully make it more clear, here’s the multiline version with a single filter

reduce x <- [1,2,3,4], 

       y <- [3,4,5], 

       x * y > 10 and x * y < 14, 

       acc: sum = 0 

do

  sum + x + y
end

You received this message because you are subscribed to a topic in the Google Groups "elixir-lang-core" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elixir-lang-core/_veg3CFQkS4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAOMhEnyO42rTZsuhGKGoBP%3DtpxdY2q5z4PyQEZh8wXraBTqzyA%40mail.gmail.com.

[Vincent Siliakus]

Op woensdag 25 februari 2015 00:41:07 UTC+1 schreef Peter Hamilton:

> I think I'm being misquoted. Let me correct :-)

Oops, sorry for the misquote.

>
> reduce n <- [1,2,3], m = 5 + 6, acc: sum = 0 do
>   sum + n*m
> end

I like this syntax, especially if 'sum = 0' allows full pattern matching. The only thing I don't understand is the 'm = 5 + 6' line. Do you propose this to have special meaning, or is it just a filter that happens to evaluate to a truthy value?

- Vincent

[Peter Hamilton]

With = you can just assign new variables. A better example would be:

iex(4)> for x <- [1,2,3], n = 4*x, y <- [3,n,5] do
...(4)>   x + y
...(4)> end

[4, 5, 6, 5, 10, 7, 6, 15, 8] 

 By defining n in the comprehension you we can use x in the definition and then use n in the collection that we unpack to y.

Since the rest of the comprehension allows full pattern matching, I would expect the acc: to allow it too. Not sure how well this would work in practice. 

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/d16b8c8c-7fc3-4aa4-af19-106da2431b9a%40googlegroups.com.

[Ben Wilson]

WRT to implementing such a comprehension, is for implemented in erlang because it couldn't be done in elixir? Or is it just for historical reasons?

On Wednesday, February 25, 2015 at 12:41:18 PM UTC-5, Peter Hamilton wrote:

With = you can just assign new variables. A better example would be:

iex(4)> for x <- [1,2,3], n = 4*x, y <- [3,n,5] do
...(4)>   x + y
...(4)> end

[4, 5, 6, 5, 10, 7, 6, 15, 8] 

 By defining n in the comprehension you we can use x in the definition and then use n in the collection that we unpack to y.

Since the rest of the comprehension allows full pattern matching, I would expect the acc: to allow it too. Not sure how well this would work in practice. 

On Wed, Feb 25, 2015, 9:08 AM Vincent Siliakus <zam...@gmail.com> wrote:

Op woensdag 25 februari 2015 00:41:07 UTC+1 schreef Peter Hamilton:
> I think I'm being misquoted. Let me correct :-)

Oops, sorry for the misquote.

>
> reduce n <- [1,2,3], m = 5 + 6, acc: sum = 0 do
>   sum + n*m
> end

I like this syntax, especially if 'sum = 0' allows full pattern matching. The only thing I don't understand is the 'm = 5 + 6' line. Do you propose this to have special meaning, or is it just a filter that happens to evaluate to a truthy value?

- Vincent

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.

[Peter Hamilton]

My guess is that for comprehensions in elixir are using the erlang for comprehensions under the hood.

Perhaps an initial implementation that would allow us to play with the syntax should pass through to a for comprehension and then call Enum.reduce afterwards. It won't be as performant but it will allow us to see what the feel of the api is.

Perhaps rewriting:

reduce x <- [1,2,3], y <- [3,4,5], acc: sum = 0 do

  sum + x*y

end

to:

interm_result = for x <- [1,2,3], y <- [3,4,5] do: {x,y}

sum = 0

Enum.reduce(interm_result, sum, fn({x,y}, sum) ->

  sum + x*y

end)

would be a good first step.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/c8304bf7-1466-4f45-8050-b354621c320c%40googlegroups.com.

[Ben Wilson]

Definitely. I was just curious if there was an expectation that a candidate reduce implementation use the same erlang roots as for. I suppose it doesn't quite matter at this point.

[José Valim]

We rely on Erlang comprehensions only if binary generators are used (as the Erlang one contains optimizations). I plan to eventually migrate the non-binary parts to Elixir. :)

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Lead Developer

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/c8304bf7-1466-4f45-8050-b354621c320c%40googlegroups.com.

[Vincent Siliakus]

Op woensdag 25 februari 2015 18:41:18 UTC+1 schreef Peter Hamilton:

With = you can just assign new variables. A better example would be:

iex(4)> for x <- [1,2,3], n = 4*x, y <- [3,n,5] do
...(4)>   x + y
...(4)> end

[4, 5, 6, 5, 10, 7, 6, 15, 8] 

 By defining n in the comprehension you we can use x in the definition and then use n in the collection that we unpack to y.

Thanks for the example, that clears it up. I'm sure you and most here realize this, but it can be a bit tricky to use filters for this. If I slightly rewrite your example to this:

for x <- [1,2,3], n = ThirdPartyLib.some_function(4), y <- [3,n,5] do
  x + y
end

you have to be really sure some_function never returns nil or false, or elements get discarded and you probably get unexpected results. Of course, nil or false in this particular example would be a problem anyway, since we're doing arithmetic, but I hope it illustrates my point.

[Wojciech Kaczmarek]

3cents:  

+1 for promoting reducers, Collectable implementations & stuff like that. It has a side effect (nomen omen) that when you do mental switch, you think in terms of reducers even in imperative languages. Which in turn helps keeping the code clean.

W.

[Robert Virding]

I hope I didn't irritate anyone, that was not the intention. It was just a result of my rather twisted sense of humour.

However, I personally think that calling it a fold fits better into a functional environment.

Robert

[Peter Hamilton]

On a lazy Saturday, I built the reduce comprehension macro. See https://github.com/hamiltop/reduce_comprehensions

First, every time I work with macros I'm amazed at how powerful they are. Take a look at the implementation. So many ideas discussed here were essentially free (the biggest one is the `acc_var` and `vars_tuple` stuff where I can use the same structure in the anonymous function definition to allow for multiple assignment).

Second, unless I'm missing something I was unable to eliminate the `for` keyword of the comprehension. Elixir doesn't support varargs and the `for` keyword is actually a special case in the parser. I believe if varargs were supported then we would be able to implement both for and reduce comprehensions entirely in elixir. Perhaps that is a worthy topic for a new thread.

Third, the implementation requires two passes through the data. This is mostly just to play around with the syntax and see if we as a community enjoy the idea of reduce comprehensions (they might be useless in practice).

Enjoy!

--

You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/61404e16-2731-478b-823a-d4eb4e3efefc%40googlegroups.com.

[José Valim]

Great work Peter!

Second, unless I'm missing something I was unable to eliminate the `for` keyword of the comprehension. Elixir doesn't support varargs and the `for` keyword is actually a special case in the parser. I believe if varargs were supported then we would be able to implement both for and reduce comprehensions entirely in elixir. Perhaps that is a worthy topic for a new thread.

One tiny correction. for is not special cased in the parser, it is special cased in the compiler. At compile time, you can have any varargs you want as long as you convert them to something else while still at compile time. You don't need to use "for" in your example, it could be any word, including thunderhorse:

reduce thunderhorse x <- list, is_atom(x) do

  ...

end

Sticking with for probably makes sense though. :)

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Lead Developer

 

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAOMhEnzYEcopudEDtCySssd9dxQEFw2PRTRuh6qfTJXCh7A-3A%40mail.gmail.com.