Introduce let and reduce qualifiers to for

[José Valim]

Hi everyone,

This is the second proposal for-let. You can find it in a gist: https://gist.github.com/josevalim/fe6b0bcc728539a5adf9b2821bd4a0f5

Please use the mailing list for comments and further discussion. Thanks for all the feedback so far!

[Dunya Kirkali]

This guide is super useful. Maybe we should use it to update the docs?

[Ben Wilson]

I believe this nicely addresses my concerns from the first proposal. Inner refactoring into functions is 100% possible, and there are no strange reassignment behaviors introduced that don't extend to the rest of the language. As for the questions outlined in the guide:

1) To Paren or not Paren: Parens optional is best IMHO.

2) Naming for let: I like let.

3) Naming for reduce: I don't _love_ reduce but I think it's the best. It is also no worse than Enum.reduce. Both are basically "for | Enum" (indicating we are dealing with a collection) "reduce" ok into a single managed value.

4) Limitations to patterns right now: I'm 100% with limiting let to `let var` or `let {tuple, of, vars}` for now. This can always be relaxed later.

All in all, very happy with where this proposal has landed, and tantalized by the future possibilities!

[eksperimental]

The proposal is very concise,
the only thing that would be problematic is the use of `reduce` for two
different things,

for <<x <- "AbCabCABc">>, x in ?a..?z, reduce: %{} do
acc -> Map.update(acc, <<x>>, 1, & &1 + 1)
end

{sum, count} =
for reduce({sum, count} = {0, 0}), i <- [1, 2, 3] do
sum = sum + i
count = count + 1
{sum, count}
end

It would lead to misunderstanding as it may not be clear which one we
are talking about when we say "use for reduce"

[José Valim]

Good point. I forgot to mention the :reduce option will be deprecated in the long term.

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/61c0d119.1c69fb81.af520.c181SMTPIN_ADDED_MISSING%40gmr-mx.google.com.

[eksperimental]

Nice,

I like better

for reduce(acc = %{}), <<x <- "AbCabCABc">>, x in ?a..?z do

Map.update(acc, <<x>>, 1, & &1 + 1)
end

over

for <<x <- "AbCabCABc">>, x in ?a..?z, reduce: %{} do
acc -> Map.update(acc, <<x>>, 1, & &1 + 1)
end

On Mon, 20 Dec 2021 19:54:01 +0100

[Stefan Chrobot]

I really like this proposal! For me it strikes the perfect balance between terseness and explicitness that I've come to enjoy in Elixir.

My votes:

- Naming: let over given; just because it's shorter,

- Do use parents: let "feels" similar to var!.

Best,

Stefan

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4LOyoAmXULJQo%2BYX4eFVJZJAoYtKHytoHujCS_kJ6AEuA%40mail.gmail.com.

[Stefan Chrobot]

I went through some of our code and one thing I'd love to see is a way to replace Enum.reduce_while with the for comprehension. So the code like this:

Enum.reduce_while(foos, {:ok, []}, fn foo, {:ok, bars} ->
  case barify(foo) do
    {:ok, bar} -> {:cont, {:ok, [bar | bars]}}
    {:error, _reason} = error -> {:halt, error}
  end
end)

Would the following even work?

for reduce(result = {:ok, []}), foo <- foos, {:ok, _} <- result do
  case barify(foo) do
    {:ok, bar} -> {{:ok, [bar | bars]}}
    {:error, _reason} = error -> {error}
  end
end

Even if it did, it's not doing a great job of communicating the intent and still potentially requires a Enum.reverse call. The intent here is "early exit with some value upon some condition or pattern mismatch".

Best,

Stefan

[José Valim]

Stefan, this would work if we include all extensions:

for reduce {status, acc} = {:ok, []}, status == :ok, foo <- foos do
  case barify(foo) do
    {:ok, bar} -> {:ok, [bar | acc]}

    {:error, _reason} = error -> error
  end
end

I am not sure if you find it any better. It is hard to do this with "let" because you don't want to include the element of when it fails.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CACzMe7aXBL1jNM_aWmJJzYOjrK%3Dtf-4%2BLPLJLpccu_G4zr0cAg%40mail.gmail.com.

[Ben Wilson]

To revisit the example situation from the original post:

```

{sections, _acc} =

for let {section_counter, lesson_counter} = {1, 1}, section <- sections do

lesson_counter = if section["reset_lesson_position"], do: 1, else: lesson_counter

{lessons, lesson_counter} = for let lesson_counter, lesson <- section["lessons"] do

{Map.put(lesson, "position", lesson_counter), lesson_counter + 1}

end

section =

section

|> Map.put("lessons", lessons)

|> Map.put("position", section_counter)

{section, {section_counter + 1, lesson_counter}}

end

```

I think that's nice! It focuses on inputs and outputs and reduces the overall line noise.

[Stefan Chrobot]

Thanks, that version would work, but agreed - not sure how big of an improvement that would be over Enum.reduce_while. Trying to think about this in a more imperative style, it seems that maybe an equivalent of break is what I'm after? Maybe something close to Ecto's Repo.transaction semantics?

for foo <- foos do
  case barify(foo) do
    {:ok, bar} -> bar
    {:error, reason} -> break(reason)
  end
end

I think this would have to work similarly to how Ecto's transactions work with Multis, that is return {:ok, bars} or {:error, reason, data_so_far}.

I have found only one usage of Enum.reduce_while in Elixir's codebase. Interestingly, the shape of the return value is always the same - a list of finished tests. https://github.com/elixir-lang/elixir/blob/main/lib/ex_unit/lib/ex_unit/runner.ex#L340

This seems like a separate problem, but maybe something to keep in mind.

Best,

Stefan

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/c1fea9e2-f47c-4236-812a-431bc7d76d62n%40googlegroups.com.

[eksperimental]

> I have found only one usage of Enum.reduce_while in Elixir's codebase.

This is because under the hood Enum.reduce_while is a call
`Enumerable.reduce/3`, so we are invoking this functions direcly.

> >>>>> https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4LOyoAmXULJQo%2BYX4eFVJZJAoYtKHytoHujCS_kJ6AEuA%40mail.gmail.com
> >>>>> <https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4LOyoAmXULJQo%2BYX4eFVJZJAoYtKHytoHujCS_kJ6AEuA%40mail.gmail.com?utm_medium=email&utm_source=footer>
> >>>>> .

> >>>>>
> >>>> --
> >>> You received this message because you are subscribed to the Google
> >>> Groups "elixir-lang-core" group.
> >>> To unsubscribe from this group and stop receiving emails from it,
> >>> send an email to elixir-lang-co...@googlegroups.com.
> >>>
> >> To view this discussion on the web visit

> >>> https://groups.google.com/d/msgid/elixir-lang-core/CACzMe7aXBL1jNM_aWmJJzYOjrK%3Dtf-4%2BLPLJLpccu_G4zr0cAg%40mail.gmail.com
> >>> <https://groups.google.com/d/msgid/elixir-lang-core/CACzMe7aXBL1jNM_aWmJJzYOjrK%3Dtf-4%2BLPLJLpccu_G4zr0cAg%40mail.gmail.com?utm_medium=email&utm_source=footer>
> >>> .

> >>>
> >> --
> > You received this message because you are subscribed to the Google
> > Groups "elixir-lang-core" group.
> > To unsubscribe from this group and stop receiving emails from it,
> > send an email to elixir-lang-co...@googlegroups.com.
> > To view this discussion on the web visit

> > https://groups.google.com/d/msgid/elixir-lang-core/c1fea9e2-f47c-4236-812a-431bc7d76d62n%40googlegroups.com
> > <https://groups.google.com/d/msgid/elixir-lang-core/c1fea9e2-f47c-4236-812a-431bc7d76d62n%40googlegroups.com?utm_medium=email&utm_source=footer>
> > .
> >
>

[eksperimental]

Not necessarily that they are replaceable,
but that pattern of :cont, :halt, :suspend is most commonly used via
Enumerable.reduce


On Mon, 20 Dec 2021 19:18:53 -0500

[José Valim]

Compared to the previous proposal, this one has been very quiet so far. :) I am unsure if it was because it was a Monday or because holidays are coming, so I am sending a ping.

ALso note I updated the proposal to not use parens around let/reduce, which seems to be the preference so far: https://gist.github.com/josevalim/fe6b0bcc728539a5adf9b2821bd4a0f5

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/61c11ffb.1c69fb81.9b607.f827SMTPIN_ADDED_MISSING%40gmr-mx.google.com.

[Stefan Chrobot]

Any thoughts about adding the early-exit semantics?

As for the parens, I was actually in favor of keeping them. "for let" and "for reduce" seem like different variants of "for", which raises the question if maybe they should be "for_let" and "for_reduce" instead.

Does this mean that the formatter will drop the parens?

Best,

Stefan

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4K9WRhyaGpXy8Ds%3DbEq%3DpapgF6w%2BRciB-PY6M6Xe5%2BoFQ%40mail.gmail.com.

[José Valim]

At the moment the early-exit semantics will be as described in the proposal. In the future, we can make the left side of "let" be a pattern, so it automatically exits as soon as it doesn't match the pattern. We could introduce non-local returns but I think they would not be accepted for the same reason implicit variables were not.

In any case, I would like to move those to a separate discussion, as there is already a lot to unpack here.

> As for the parens, I was actually in favor of keeping them.

I am mostly reflecting the current preferences but there is one argument for not having parens, which is if we were to introduce async. Async doesn't have its own arguments, so it would be `for async(), x <- collection`, which doesn't look nice. for_let and for_reduce may be a discussion worth having though.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CACzMe7bYaEzaeeGpiKfux7K%2BLGOGMQS5cdFejqq5%2BEPu056uWA%40mail.gmail.com.

[Bruce Tate]

Minus one for me on for_let and for_reduce naming as this style of concept composition limits the number of concepts we can add later. I like the rest of the latest proposal as stated, and I like the idea of deferring the match-while semantics. 

-bt

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4LTqZ223yVFNgTVgm%3DS4OC0p7cTyxMUYysZnFvdnMfh%3Dg%40mail.gmail.com.

--

Regards,

Bruce Tate

CEO

Groxio, LLC.

512.799.9366

br...@grox.io

grox.io

[Adam Lancaster]

The one thing I have in the back of my mind is that what I enjoy about elixir is the explicitness. This feels like it might trick people into thinking that there is mutable data or something, which isn't the case. 

In the examples given I can't see how they are different from using reduce with a tuple as an accumulator, eg:

```

for i <- [1, 2, 3], reduce: {0, 0}  do

  {doubled, sum} -> {i * 2, i + sum}

end

```

I'm possibly missing something but I worry that it hides what's actually happening!

Best

Adam

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAFXvW-66yKSPBxTC7b8paYOWEnD0DZDc65k-BUR4wX__XdV2%3Dg%40mail.gmail.com.

[José Valim]

In the examples given I can't see how they are different from using reduce with a tuple as an accumulator, eg:

```

for i <- [1, 2, 3], reduce: {0, 0}  do

  {doubled, sum} -> {i * 2, i + sum}

end

```

There is some confusion here. "reduce" cannot return the modified list, so the example above is not doing what you expect it to. However, let's assume you want to write this instead:

for i <- [1, 2, 3], reduce: {0, 0}  do

  {count, sum} -> {count + 1, i + sum}

end

Then you are right, they are not different. In the reduce case, it is a different syntax to achieve the same thing. So you may ask: why change?

I think there are two benefits to the proposed syntax:

1. Collocating variable declaration with initialization

In your example, you need pass the initial value in one place and then the names they receive is elsewhere:

for i <- [1, 2, 3], reduce: {0, 0}  do

  {count, sum} -> {count + 1, i + sum}
end

For example, we could change it to this to address this issue:

for i <- [1, 2, 3], reduce: {count, sum} = {0, 0}  do
  {count + 1, i + sum}
end

 

2. By declaring the variables that are part of reduce prior to the generators, they can be used as part of the filters. So we can further change the code above to this:

for reduce {count, sum} = {0, 0}, i <- [1, 2, 3]do
  {count + 1, i + sum}
end

And that would allow us to, for example, count only the first 100:

for reduce {count, sum} = {0, 0}, count < 100, i <- [1, 2, 3] do
  {count + 1, i + sum}
end

Furthermore, I would say declaring reduce upfront is better, because you don't need to read all generators and filters to figure out what the comprehension will actually return.

So there is more we can do with the new syntax, but from the point of view of inputs/outputs, they are equivalent. The implicit variable assignment is gone. The only downside of the proposed syntax compared to the current one is that the current one makes it easier if you want to have multiple clauses. But it is not such a common case and you could use a "case ... do" as well.

In other words, even if we are not to add "for let", I would say this version of reduce is superior and it should be considered in itself.

[Adam Lancaster]

Thank you for explaining that makes sense.

My final reservation would be that everywhere else in Elixir when you see `=` it's a pattern match. Does this break that? Would this (contrived) example be valid?:

for reduce {%{start: count}, sum} = {%{start: 0}, 0}, count < 100, i <- [1, 2, 3] do

  {count + 1, i + sum}

end

Best

Adam

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4LmhgA%3De9_vZ8_1eyv3gjsUsArYZH31388s-%3D%2BY-uUM5Q%40mail.gmail.com.

[José Valim]

 

Does this break that? Would this (contrived) example be valid?:

for reduce {%{start: count}, sum} = {%{start: 0}, 0}, count < 100, i <- [1, 2, 3] do

  {count + 1, i + sum}

end

It depends. It is still a pattern match but an initial version would allow only a subset of patterns. We should be able to relax it, but I don't want to start with that so we can collect more feedback. And, to be clear, if we do so, you should write:

for reduce {%{start: count}, sum} = {%{start: 0}, 0}, count < 100, i <- [1, 2, 3] do

  {%{start: count + 1}, i + sum}

end

[Greg Vaughn]

I'm a vote in favor of the proposal. I think I have a slight preference for the parens. But it's very slight, and I'm not sure I wouldn't change my mind after using it for a while.

I am against the notion of `for_let` and `for_reduce`. I see that as using up more of the special forms "namespace".

I'm slightly disappointed that the old style `reduce:` option will be deprecated (because I have old code I'll have to update) however, the change does make things more consistent and I appreciate the thought about future extensions to for comprehensions.

-Greg Vaughn

PS. I just watched your latest Twitch stream. I wish I could have been there live to hear you read the dictionary and to offer obscure English words to the bikeshed discussion :-D

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4K9WRhyaGpXy8Ds%3DbEq%3DpapgF6w%2BRciB-PY6M6Xe5%2BoFQ%40mail.gmail.com.

[Christopher Keele]

Guides Feedback

These look excellent, should make the power of for much more accessible.

let Feedback

I like the functionality of this proposal much more!

I will say, I'm still not a fan of using new qualifiers. I'm not sure why we need to introduce a new special form for this, since we can do whatever we need to in the macro? What's wrong with:

> for count = 0, count < 5, x <- element do {x, count + 1} end

Perfectly matches traditional for loops, and pattern matching in function arguments, with all the familiar errors/warning possibly emitted from them,  while keeping the bodies identically functionally-refactorable. No lets needed.

Re: for reduce, since for already has idiomatic keyword arguments to further modify functionality, isn't :reduce is doing fine as is? We'd just document how it interacts with for-bindings.

> Furthermore, the introduction of let and reduce qualifiers opens up the option for new qualifiers in the future, such as for async that is built on top of Task.async_stream/3.

>

> Async doesn't have its own arguments

Similarly, I'm unsure what's wrong with add an async: true keyword argument. It's just behaviour configuration, not something we need to bring to the forefront as a pattern match?

Replies

> I am against the notion of `for_let` and `for_reduce`. I see that as using up more of the special forms "namespace".

Agreed, mirroring my misgivings about "for let" and "for reduce".

[Stefan Chrobot]

I'm too voting in favor of the proposal in the current form. Unsure about the parens, but looking forward to some way of replacing reduce_while with some version of the for.

Best,

Stefan

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/1990bd4f-ce14-43cb-958a-5e261a9a3d22n%40googlegroups.com.

[Zach Daniel]

I really like this, and would be happy with it as is I have one thought though, with the way that we are already doing a bit of “magic” (I don’t mean that in a negative way) to map the “let” variable to the second element of the tuple, could we support multiple assignments without the tuple in the let? Something like this (hard to type code on the phone )

for let x = 10, y = 12, foo <- list do

{result, x, y}

end

Sent via Superhuman iOS

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CACzMe7aQ5h8UEsKtqW5_rNLQQqjzWt-t4oR_WW8-sRzT808O4w%40mail.gmail.com.

[Zach Daniel]

for let x = 10, let y = 12, foo <- list do

{result, x, y}

end

Sent via Superhuman iOS

On Tue, Dec 21 2021 at 4:35 PM, Zach Daniel <zachary....@gmail.com> wrote:

I really like this, and would be happy with it as is I have one thought though, with the way that we are already doing a bit of “magic” (I don’t mean that in a negative way) to map the “let” variable to the second element of the tuple, could we support multiple assignments without the tuple in the let? Something like this (hard to type code on the phone )

for let x = 10, y = 12, foo <- list do

{result, x, y}

end

Sent via Superhuman iOS

[eksperimental]

On Tue, 21 Dec 2021 14:03:49 +0100
José Valim <jose....@dashbit.co> wrote:

> 2. By declaring the variables that are part of reduce prior to the
> generators, they can be used as part of the filters. So we can further
> change the code above to this:

So, the reduce() and let() part is going to be accepted only at the
beginning or we will be able to call them wherever we find suitable?

[Sabiwara Yukichi]

I like this new proposal a lot, it brings the power of map_reduce to comprehensions without any need for new concepts to learn.

I was just wondering how it could be used from EEx/Phoenix templates, looking at the previous example from Chris:

    <%= for item <- items, let: {highlighted? = false} do

      ...

      <% highlighted? = not highlighted? %>

    <% end %>

The let/reduce alternatives only offer the option to discard or not the list, but will always return the accumulator (which forces to pattern-match on a tuple if we just want the list). Maybe we could have an option to just discard the accumulator and return the list (as in the template use case), or maybe even another option to discard both and return :ok (to enable Enum.each/2 like comprehensions)? Not sure what the API could look like.

Le mar. 21 déc. 2021 à 03:12, José Valim <jose....@dashbit.co> a écrit :

Hi everyone,

This is the second proposal for-let. You can find it in a gist: https://gist.github.com/josevalim/fe6b0bcc728539a5adf9b2821bd4a0f5

Please use the mailing list for comments and further discussion. Thanks for all the feedback so far!

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4KHhMO3cFzf3L%3D7e1EJbC8hu5bLFUA%2BTRSkb9iDGozUSQ%40mail.gmail.com.

[José Valim]

Thanks everybody for the feedback. Replies below!

> I will say, I'm still not a fan of using new qualifiers. I'm not sure why we need to introduce a new special form for this,

> since we can do whatever we need to in the macro? What's wrong with: for count = 0, count < 5, x <- element do {x, count + 1} end

I did consider this. However, using "count = ... " as a filter anywhere does not change the return type of the comprehension. Why does setting the variable at the beginning change the return type? Given it has a strong impact on the return type, I think we need a clearer indicator.

Aso, "var = expr" in a comprehension means that, if expr is false or nil, then no further code is executed. If we were to keep the semantics, then I couldn't set an initial accumulator to nil.

In other words, the semantics are just too different for us to rely on the existing behaviour.

> I really like this, and would be happy with it as is I have one thought though, with the way

> that we are already doing a bit of “magic” (I don’t mean that in a negative way) to map the “let”

> variable to the second element of the tuple, could we support multiple assignments without the

> tuple in the let?

I also considered this and I think the "multiple lets" and "multiple reduces" could get confusing. Can I have both? Can I declare them anywhere? The answer is no. Given you can't combine them and only use them at the beginning, it feels like having only one is the more appropriate choice.

> I was just wondering how it could be used from EEx/Phoenix templates, looking at the previous example from Chris:

Phoenix will most likely need to use the implicit accumulator to make this work. Then it becomes a question to ask the Phoenix team. However, given it is exclusive to the template, it is probably fine to be implicit.

[Zach Daniel]

I also considered this and I think the "multiple lets" and "multiple reduces" could get confusing. Can I have both? Can I declare them anywhere? The answer is no. Given you can't combine them and only use them at the beginning, it feels like having only one is the more appropriate choice.

Yeah, that makes sense. I'm all for it. Especially doing this year's advent of code, this would have cleaned up *tons* of code, so I'm very excited about this proposal.

Sent via Superhuman

--

You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4%2BmRv0%2BptoVoKSN11y_C%2B56QP1vKBqXVdDFvoZQpqh7qQ%40mail.gmail.com.