Introduce :let in for-comprehensions

217 views
Skip to first unread message

José Valim

unread,
Dec 16, 2021, 10:02:49 AM12/16/21
to elixir-lang-core

Note: This proposal contains images and rich text that may not display correctly in your email. If so, you can read this proposal in a gist.

There is prior art in languages like Common Lisp, Haskell, and even in C# with LINQ on having very powerful comprehensions as part of the language. While Elixir comprehensions are already very expressive, allowing you to map, filter, reduce, and collect over multiple enumerables at the same time, it is still not capable of expressing other constructs, such as map_reduce.

The challenge here is how to continue adding more expressive power to comprehensions without making the API feel massive. That's why, 7 years after v1.0, only two new options have been added to comprehensions, :uniq and :reduce, to a total of 3 (:into, :uniq, and :reduce).

Imperative loops

I have been on the record a couple times saying that, while many problems are more cleanly solved with recursion, there is a category of problems that are much more elegant with imperative loops. One of those problems have been described in the "nested-data-structures-traversal" repository, with solutions available in many different languages. Please read the problem statement in said repository, as I will assume from now on that you are familiar with it.

Personally speaking, the most concise and clear solution is the Python one, which I reproduce here:

section_counter = 1
lesson_counter = 1

for section in sections:
    if section["reset_lesson_position"]:
        lesson_counter = 1

    section["position"] = section_counter
    section_counter += 1

    for lesson in section["lessons"]:
        lesson["position"] = lesson_counter
        lesson_counter += 1

There are many things that make this solution clear:

  • Reassignment
  • Mutability
  • Sensitive whitespace

Let's compare it with the Elixir solution I wrote and personally prefer. I am pasting an image below which highlights certain aspects:

Screenshot 2021-12-13 at 10 02 48

  • Lack of reassignment: in Elixir, we can't reassign variables, we can only rebind them. The difference is, when you do var = some_value inside a if, for, etc, the value won't "leak" to the outer scope. This implies two things in the snippet above:

    1. We need to use Enum.map_reduce/3 and pass the state in and out (highlighted in red)
    2. When resetting the lesson counter, we need both sides of the conditional (hihhlighted in yellow)

  • Lack of mutability: even though we set the lesson counter inside the inner map_reduce, we still need to update the lesson inside the session (highlighted in green)

  • Lack of sensitive whitespace: we have two additional lines with end in them (highlighted in blue)

As you can see, do-end blocks add very litte noise to the final solution compared to sensitive whitespace. In fact, the only reason I brought it up is so we can confidently discard it from the discussion from now on. And also because there is zero chance of the language suddenly becoming whitespace sensitive.

There is also zero chance of us introducing reassignment and making mutability first class in Elixir too. The reason for this is because we all agree that, the majority of the time, lack of reassignment and lack of mutability are features that make our code more readable and understandable in the long term. The snippet above is one of the few examples where we are on the wrong end of the trade-offs.

Therefore, how can we move forward?

Comprehensions

Comprehensions in Elixir have always been a syntax sugar to more complex data-structure traversals. Do you want to have the cartesian product between all points in x and y? You could write this:

Enum.flat_map(x, fn i ->
  Enum.map(y, fn j -> {i, j} end)
end)

Or with a comprehension:

for i <- x, j <- y, do: {i, j}

Or maybe you want to brute force your way into finding Pythagorean Triples?

Enum.flat_map(1..20, fn a ->
  Enum.flat_map(1..20, fn b ->
    1..20
    |> Enum.filter(fn c -> a*a + b*b == c*c end)
    |> Enum.map(fn c -> {a, b, c} end)
  end)
end)

Or with a comprehension:

for a <- 1..20,
    b <- 1..20,
    c <- 1..20,
    a*a + b*b == c*c,
    do: {a, b, c}

There is no question the comprehensions are more concise and clearer, once you understand their basic syntax elements (which are, at this point, common to many languages).

As mentioned in the introduction, we can express map, filter, reduce, and collect inside comprehensions. But how can we represent map_reduce in a clear and concise way?

The :map_reduce option

Since we have :reduce in comprehensions, we could introduce :map_reduce. The solution above would look like this:

{sections, _acc} =
  for section <- sections, map_reduce: {1, 1} do
    {section_counter, lesson_counter} ->
      lesson_counter = if section["reset_lesson_position"], do: 1, else: lesson_counter

      {lessons, lesson_counter} =
        for lesson <- section["lessons"], map_reduce: lesson_counter do
          lesson_counter ->
            {Map.put(lesson, "position", lesson_counter), lesson_counter + 1}
        end

      section =
        section
        |> Map.put("lessons", lessons)
        |> Map.put("position", section_counter)

      {section, {section_counter + 1, lesson_counter}}
  end

While there is a bit less noise compared to the original solution, the reduction of noise mostly happened by the removal of modules names and a few tokens, such as fn, (, and ). In terms of implementation, there is still a lot of book keeping required to manage the variables. Can we do better?

Introducing :let

Our goal is to declare variables that are automatically looped within the comprehension. So let's introduce a new option that does exactly that: :let. :let expects one or a tuple of variables that will be reused across the comprehension. At the end, :let returns a tuple with the comprehension elements and the let variables.

Here is how the solution would look like:

section_counter = 1
lesson_counter = 1

{sections, _} =
  for section <- sections,
      let: {section_counter, lesson_counter} do
    lesson_counter = if section["reset_lesson_position"], do: 1, else: lesson_counter

    {lessons, lesson_counter} =
      for lesson <- section["lessons"], let: lesson_counter do
        lesson = Map.put(lesson, "position", lesson_counter)
        lesson_counter = lesson_counter + 1
        lesson
      end

    section =
      section
      |> Map.put("lessons", lessons)
      |> Map.put("position", section_counter)

    section_counter = section_counter + 1
    section
  end

The :let option automatically takes care of passing the variables across the comprehension, considerably cutting down the noise, without introducing any mutability into the language. At the end, for+:let returns the result of the comprehension plus the :let variables wrapped in a tuple.

Extensions

Here are some extensions to the proposal above. Not all of them might be available on the initial implementation.

Let initialization

You can also initialize the variables within let for convenience:

{sections, _} =
  for section <- sections,
      let: {section_counter = 1, lesson_counter = 1} do

This should be available in the initial implementation.

:reduce vs :let

With :let, :reduce becomes somewhat redundant. For example, Enum.group_by/2 could be written as:

for {k, v} <- Enum.reverse(list), reduce: %{} do
  acc -> Map.update(acc, k, [v], &[v | &1])
end

with :let:

{_, acc} =
  for {k, v} <- Enum.reverse(list), let: acc = %{} do
    acc = Map.update(acc, k, [v], &[v | &1])
  end

The difference, however, is that :let returns the collection, while :reduce does not. While the Elixir compiler could be smart enough to optimize away building the collection in the :let case if we don't use it, we may want to keep both :let and :reduce options for clarity. If this is the case, I propose to align the syntaxes such that :reduce uses the same semantics as :let. The only difference is the return type:

for {k, v} <- Enum.reverse(list), reduce: acc = %{} do
  acc = Map.update(acc, k, [v], &[v | &1])
end

This can be done in a backwards compatible fashion.

after

When you look at our solution to the problem using let, we had to introduce temporary variables in order to update our let variables:

{lessons, lesson_counter} =
  for lesson <- section["lessons"], let: lesson_counter do
    lesson = Map.put(lesson, "position", lesson_counter)
    lesson_counter = lesson_counter + 1
    lesson
  end

One extension is to add after to the comprehensions, which are computed after the result is returned:

{lessons, lesson_counter} =
  for lesson <- section["lessons"], let: lesson_counter do
    Map.put(lesson, "position", lesson_counter)
  after
    lesson_counter = lesson_counter + 1
  end

This does not need to be part of the initial implementation.

Summary

Feedback on the proposal and extensions is welcome!

Louis Pilfold

unread,
Dec 16, 2021, 10:46:45 AM12/16/21
to elixir-lang-core
Heya

My initial impression is that this is quite a large departure from the semantics of all other constructs in the language, and the usual rules for reading Elixir no longer apply. The "return value" of a loop block is no longer the final line of the block, it could be spread throughout it.

To what extent can one use these mutable values with other language features?

Is this permitted?
for lesson <- lessons, let: lesson_counter do
  if lesson.active? do

    lesson_counter = lesson_counter + 1
  end

end
Or this?
for lesson <- lessons, let: lesson_counter do
  Enum.each(lesson.sessions, fn(_) ->

    lesson_counter = lesson_counter + 1
  end)
end
If the above snippet with an anonymous function does not work, I think that may be quite surprising to newcomers as both `Enum.*` and `for` would work in languages which have conventional mutable variables (which they will likely be mentally using as a reference).

Overall I don't feel this added complexity and potential for confusion is worth the benefits here. This brings no more expressive power to the language, and personally I don't have any issues with the "before" examples here using existing language features.
Having said that, I'm not a big user of `for`, so the opinions of others may be more useful here!

Cheers,
Louis


--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4JjyZ2EUcYm1TA747pP2pTgDAD_%3DW%2BM9mSizFHJXFfqnQ%40mail.gmail.com.

Paul Clegg

unread,
Dec 16, 2021, 11:12:30 AM12/16/21
to elixir-l...@googlegroups.com
My gut reaction is the same as Louis'.  Looking at the problem statement, the solution, to me, is ultimately a map over the array.  The only "problem" is maintaining some state between elements, which is managed well with the map_reduce.  All the proposed variations just seem "overworked" to me -- the original Elixir solution seemed more concise than either of the comprehension proposals.  Also like Louis, I rarely use the "for" construction -- I will more likely use Enum.each or Enum.with_index or Enum.map to iterate over a list, rather than "for".  Maybe I should use "for" more often, but it's not my go-to iterator.  :)

...Paul




To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CABu8xFAMx3hTfFbu%2B%2BN9XXnG5sEJ4zTSXdugSa96-u9uEGq%2B5g%40mail.gmail.com.

Aleksei Matiushkin

unread,
Dec 16, 2021, 11:42:54 AM12/16/21
to elixir-l...@googlegroups.com
I do not have any issues with the existing syntax as well. I don’t think `let:` is somewhat I am going to use on daily basis, but I also don’t see any harm while the old syntax for `reduce` stays.

While we are on topic, I’d share another thing I really miss in `for` comprehensions: early returns aka `reduce_while` aka `take`. When one wants to use the power of comprehension to find the first element meeting some requirements, `for` cannot be used because it will greedily iterate through the tail of the iterations. Consider the following example:

for i <- 0..1_000, j <- 0..1_000, value = some_get_fun(some_matrix, i, j), not is_nil(value), do: value

If I want to find a first non-nil value, I have to either go through all the 1M iterations and then pipe the result to hd() or use `throw value` and wrap the call into `try/catch`. It’d be great to have somewhat like `take: 1` option to stop evaluation immediately after it has been emitted into `do:` clause.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAD3kWz99qSby_7PjBT%3D68ztzmX-2tq%2B-xV7RXL_ze4ZXsaFp7w%40mail.gmail.com.


--
Aleksei Matiushkin, Software Engineer - R&D
 
 


8 Devonshire Square, London, EC2M 4PL, United Kingdom
Torre Mapfre, Planta 22, Marina, 16-18, 08005 Barcelona, Spain

  








LinkedIn    Twitter    YouTube
 
Kantox Limited is a UK private company with registered company number 07657495 and registered address at 8 Devonshire Square, London EC2M 4PL, United Kingdom. We are authorised with the UK Financial Conduct Authority (FCA) under the Payment Service Regulation 2017 as a Payments Institution (FRN 580343) for the provision of payment services and with HMRC as a Money Service Business Registration No.12641987.
Kantox European Union, S.L.  is a Spanish private company with tax ID number B67369371 and registered address at Torre Mapfre, Planta 22, Marina, 16-18, 08005 Barcelona, Spain. Kantox is authorized by the Bank of Spain, with registration number 6890, which is the supervisor of the Spanish banking system along with the European Central Bank. Additionally, we are supervised by SEPBLAC, the Supervisory Authority for the prevention of money laundering and terrorist financing in Spain.
KANTOX is the Controller for the processing of data in accordance with the GDPR and LOPDGDD for the purpose of maintaining a commercial relationship. You may exercise your rights of access and rectification, portability, restriction and opposition by writing to KANTOX to the email: gd...@kantox.com. You have your right to make a complaint at www.aepd.es.  

Ben Wilson

unread,
Dec 16, 2021, 11:54:50 AM12/16/21
to elixir-lang-core
I am with Louis and Paul so far I think. I won't repeat their comments but I think I can extend the issue by pointing out that this breaks refactoring for the inner contents of `for`. Previously, if you have:

```
  for lesson <- section["lessons"], reduce: 0 do
    counter ->
      # complex multi-line-thing using the lesson and counter
  end 
```

I can refactor this into:

```
  for lesson <- section["lessons"], reduce: 0 do
    counter ->
      complex_operation(lesson, counter)
  end 

  def complex_thing(lesson, counter) do
    # complex multi-line-thing using the lesson and counter
  end
```

And everything just works, as is normal in Elixir code. The proposed changes would (as far as I can see) break this and that feels very unexpected and foreign.

I sympathize with the problem space, but so far it's a -1 for me on this particular proposed improvement.

- Ben

José Valim

unread,
Dec 16, 2021, 1:10:22 PM12/16/21
to elixir-lang-core
Hi everyone,

Thanks for the input so far. I have one favor to ask: could everyone please think carefully about the problem statement?

Put yourself into the shoes of someone starting with Elixir and having to solve this problem. How many concepts do you have to learn to get this (allegedly simple) task done? I am aware most people in this mailing list are comfortable with the Enum.map_reduce/3 solution presented, but we need to be careful to not fall into the trap of "I've climbed this mountain, I bet others can too!". In the linked repository, there are solutions in many languages, and Elixir (and most functional programming languages) are worse than most of the mainstream languages in terms of size, noise, and required concepts.

I am NOT saying we need to sacrifice the things that make Elixir be Elixir. I am NOT proposing to "dumb down" the language. There are valid points against this proposal! But I think there would be a strong dissonance if we can't agree on the problem statement and on how noisy and mechanical the current functional solution looks. :)

Some comments/replies below (I am using bold for questions/previous remarks).

---

> To what extent can one use these mutable values with other language features?

It would only be at the comprehension root. You are correct there would still cause confusion as people may expect it to apply in other places.

> Overall I don't feel this added complexity and potential for confusion is worth the benefits here. This brings no more expressive power to the language.

The goal was precisely to not add more expressive power (specifically, not add mutation). :D

This is a bit unrelated and I can't remember exactly where I read/heard it but there is a quote along the lines: "Expressive power is a good measure for language capabilities but it is not a good measure for language features". Otherwise, we would all be writing fun(arg1)(arg2) instead of fun(arg1, arg2). :)

>  It’d be great to have somewhat like `take: 1` option to stop evaluation immediately after it has been emitted into `do:` clause.

There is already an open issue for the :take option.

> I won't repeat their comments but I think I can extend the issue by pointing out that this breaks refactoring for the inner contents of `for`

Correct. This is personally my biggest concern against this proposal, especially because we have removed features in the past (imperative assignments for conditionals) because they broke this property.


--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/22703d4b-60cb-4b0e-83d2-4a122f9147afn%40googlegroups.com.

Louis Pilfold

unread,
Dec 16, 2021, 1:54:37 PM12/16/21
to elixir-lang-core
Heya

> Put yourself into the shoes of someone starting with Elixir and having to solve this problem. How many concepts do you have to learn to get this (allegedly simple) task done?

This is the point of view from which I wrote my email- I think this proposal increases the amount of knowledge a newcomer would need to have in order to understand this code.

They would both be unable to use their pre-existing understanding of mutable variables (as the rules are very different), and there would be another set of rules to learn that do not exist elsewhere in Elixir. My hunch is that this feature is one for power-users, not one for newcomers.

> How many concepts do you have to learn to get this (allegedly simple) task done?

Adding a new concept to learn only reduces the number of concepts needed to learn in order to perform a new task if it can replace several of the others. This concept can only be applied within `for`, and then only in a very restricted fashion, so I don't think it would enable new Elixir developers to skip over learning Elixir as it is today. Because of this I believe it's now N+1 concepts to learn.

Cheers,
Louis

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4KOF759bonEQ-5zX3_ebcO7y8ySOqS3_oeDcMdc-6Dwgw%40mail.gmail.com.

José Valim

unread,
Dec 16, 2021, 2:18:10 PM12/16/21
to elixir-lang-core
> They would both be unable to use their pre-existing understanding of mutable variables (as the rules are very different), and there would be another set of rules to learn that do not exist elsewhere in Elixir. My hunch is that this feature is one for power-users, not one for newcomers.

Perfect, thank you Louis!

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CABu8xFCiT_egSe-PmUmoQR%2BmQDmTigvJfF_wKman9_FQKrqYuA%40mail.gmail.com.

Paul Schoenfelder

unread,
Dec 16, 2021, 2:18:42 PM12/16/21
to 'Justin Wood' via elixir-lang-core
I just want to +1 what Louis has said. Overall, this feels like it makes things more complex rather than less, particularly because of the limited scope that the feature applies to, which makes it difficult to fit into one’s intuition as they learn the language. 

While the original example is more complex to learn up front for someone new to Elixir and FP, the tools you are learning to use there are applicable to all problems you face, and it builds towards an intuition that will serve you well as you gain experience with the language. This feature feels out of place from that perspective, at least in my opinion.

Have you run this by anyone you know that hasn’t really used the language? I’d be very interested to get the opinion of someone that doesn’t have any preconceived notions of what fits or doesn’t in Elixir, and can evaluate what option is easiest to learn.

Paul

Chris McCord

unread,
Dec 16, 2021, 2:50:40 PM12/16/21
to elixir-l...@googlegroups.com
I was skeptical at first as well until I started thinking more about how convoluted some scenarios are today, and I think we on this mailing list especially are all likely a bit stuck in our ways. If you think about it from the beginner perspective, there is a ton of ceremony for certain operations, and some even today I have to go lookup to do basic things after almost 9 years of elixir. Consider the "make every other item highlighted" scenario. You either have to use `Enum.with_index` and `rem(index, 2) == 0`, which already is introducing two new APIs to the user, or you need to use Enum.zip with Stream.cycle:

    <%= for {item, highlighted?} <- Enum.zip(items, Stream.cycle([true, false])) %>

This looks reasonable, but is extremely non intuitive for newcomers, and even I have to think hard or lookup with I've done previously to remember it zip + cycle. Consider the alternative:

    <%= for item <- items, let: {highlighted? = false} do
      ...
      <% highlighted? = not highlighted? %>
    <% end %>

This will also help cut down on the number of unnecessary traversals folks need to make in general, but performance is not close the main benefits here. While this at first felt like mutation to me and unlike anything we do in elixir, in my mind it actually aligns really well with the way variable rebinding already works – it is simply being extended to for bodies.

+1 from me.


The :let option automatically takes care of passing the variables across the comprehension, considerably cutting down the noise, without introducing any mutability into the language. At the end, for+:let returns the result of the comprehension plus the :letvariables wrapped in a tuple.

Extensions

Here are some extensions to the proposal above. Not all of them might be available on the initial implementation.

Let initialization

You can also initialize the variables within let for convenience:

{sections, _} =
  for section <- sections,
      let: {section_counter = 1, lesson_counter = 1} do

This should be available in the initial implementation.

:reduce vs :let

With :let, :reduce becomes somewhat redundant. For example, Enum.group_by/2 could be written as:

for {k, v} <- Enum.reverse(list), reduce: %{} do
  acc -> Map.update(acc, k, [v], &[v | &1])
end

with :let:

{_, acc} =
  for {k, v} <- Enum.reverse(list), let: acc = %{} do
    acc = Map.update(acc, k, [v], &[v | &1])
  end

The difference, however, is that :let returns the collection, while :reduce does not. While the Elixir compiler could be smart enough to optimize away building the collection in the :let case if we don't use it, we may want to keep both :let and :reduceoptions for clarity. If this is the case, I propose to align the syntaxes such that :reduce uses the same semantics as :let. The only difference is the return type:

for {k, v} <- Enum.reverse(list), reduce: acc = %{} do
  acc = Map.update(acc, k, [v], &[v | &1])
end

This can be done in a backwards compatible fashion.

after

When you look at our solution to the problem using let, we had to introduce temporary variables in order to update our let variables:

{lessons, lesson_counter} =
  for lesson <- section["lessons"], let: lesson_counter do
    lesson = Map.put(lesson, "position", lesson_counter)
    lesson_counter = lesson_counter + 1
    lesson
  end

One extension is to add after to the comprehensions, which are computed after the result is returned:

{lessons, lesson_counter} =
  for lesson <- section["lessons"], let: lesson_counter do
    Map.put(lesson, "position", lesson_counter)
  after
    lesson_counter = lesson_counter + 1
  end

This does not need to be part of the initial implementation.

Summary

Feedback on the proposal and extensions is welcome!

José Valim

unread,
Dec 16, 2021, 3:15:32 PM12/16/21
to elixir-lang-core
> Have you run this by anyone you know that hasn’t really used the language? I’d be very interested to get the opinion of someone that doesn’t have any preconceived notions of what fits or doesn’t in Elixir, and can evaluate what option is easiest to learn.

I did discuss this during the Twitch streams and that has a reasonable amount of people either unfamiliar or getting started with Elixir and the reaction was largely positive.

I understand the point that this is something very specific to "for" but "for" is already very specific in itself. Constructs like <- acting as a generator only works on for but it is also very common in a huge spectrum of languages (but it might use another operator). So I partially think that "for" is the perfect place to add this behaviour, without leaking to the rest of the language, but there is a question on how to best communicate this intent.

There is also the point that most Elixir getting started materials likely don't talk about "for", so it is probably introduced too late, and most likely after you make people grok recursion. While most imperative languages introduce the concepts behind for and loops in very early lessons.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/b55240b6-5eb0-4cbc-a1cf-5016a130e1e0%40www.fastmail.com.

Amos King - Binary Noggin

unread,
Dec 16, 2021, 3:21:19 PM12/16/21
to elixir-l...@googlegroups.com
I think the `let` feature might have a big impact on the way that people write code in Elixir. I don't know if that will be positive or negative. I lean more toward expecting it to not be great. It feels more imperative. Like setting a variable before a `for` loop in Java, C, ...

That all said I have a small thought about the syntax if it is decided that it moves forward.
Looking at the examples I think that defining something outside the loop and mutating it in the loop feels awkward compared to how I have become accustomed to Elixir and other functional languages.

I actually think that Chris McCord's example where the `let` is defining the initial value is much better.

What about using a keyword syntax there, and not allowing the use of variable declared above the `let`?

for {k, v} <- Enum.reverse(list), reduce: acc: %{} do
  acc = Map.update(acc, k, [v], &[v | &1])
end

Amos King, CEO

      

573-263-2278 am...@binarynoggin.com




To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/657C7140-D8C2-4FF6-8616-3CD92400F1EB%40chrismccord.com.

Stefan Chrobot

unread,
Dec 16, 2021, 4:17:37 PM12/16/21
to elixir-l...@googlegroups.com
How about this:

for section <- sections,
  let: section_counter <- 1,
  let: lesson_counter <- 1 do


  lesson_counter = if section["reset_lesson_position"], do: 1, else: lesson_counter

  {lessons, lesson_counter} =
    for lesson <- section["lessons"],
        let: lesson_counter <- lesson_counter do

      {Map.put(lesson, "position", lesson_counter), lesson_counter + 1}
    end

  section =
    section
    |> Map.put("lessons", lessons)
    |> Map.put("position", section_counter)

  {section, section_counter + 1, lesson_counter}
end

Rationale:
- "section <- sections" already rebinds the section variable on each loop, so feels like a natural way to signal that something is changing on each iteration,
- "let: <variable name> <- <initial value>" declares a variable that will be manually rebound,
- "{section, section_counter + 1, lesson_counter}" - explicitly returns the usual value and the rebound values; fails if the tuple does not match the "let"s

I'm not exactly happy about "{lessons, lesson_counter} =" - this can only work if we return the whole tuple from the for. But maybe that's better for consistency?


Best,
Stefan

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAJr6D4S3WfHgmryrz9o%3D1AFwebbLRfakUN6Vvi%3D5w8knYjVnPw%40mail.gmail.com.

José Valim

unread,
Dec 16, 2021, 5:17:31 PM12/16/21
to elixir-lang-core
Thanks everyone,

I am discarding the proposal for now and I will resubmit a new one next week. At this point, it is clear the "imperative assignment" is a big departure and any new proposal must not include that.

Just one last point. Let's change the Pythagorean Triplets example to store values in Map. We can do this:
for a <- 1..20,
    b <- 1..20,
    c <- 1..20,
    a*a + b*b == c*c
,
    reduce: %{} do
  acc -> Map.put(acc, {a, b}, c)
end
The whole  "reduce: %{} do acc ->" is a construct that you have to learn and, although it plays by the language syntax rules, it is quite specific to "for". Maybe your answer is, "that's why I don't use for", but that's partially the point: we already have for, it has its own rules, but they are verbose, not frequently used, while still having a lot of potential inside them. As much potential as "for" in most languages.

How can we teach someone how to do a map_reduce without telling them about map_reduce? I don't believe it is possible today without this additional cognitive load. You do need to learn where it comes from at some point, but we don't learn how to paint by first learning the name of all colors.

Some more replies below (quotations in bold).

> let: section_counter <- 1, let: lesson_counter <- 1

My concern about this is that `<-` in for means extracting something from the collection, so giving it another meaning inside an option can be quite confusing.

> for {k, v} <- Enum.reverse(list), reduce: acc: %{} do

My very early prototypes (I have been working on this for several weeks now) used keyword lists but there is one big issue. Variables in Elixir are identified by their name and their context (which is important for variable hygienes in macros). A keyword list has atom as keys and they don't carry the context, which means we can't use atoms (keys in keyword lists) to express variables.


Bruce Tate
unread,
Dec 16, 2021, 5:49:39 PM12/16/21
to elixir-l...@googlegroups.com
I love this example. I am training new Elixir users in  LiveView and OTP, and there are a few burs that don't overly taint Elixir, but could smooth the learning process. This is one of them. One of the techniques I teach is naming concepts, and this is a great opportunity to name a concept. 

Big +1 from me. 

-bt


To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/657C7140-D8C2-4FF6-8616-3CD92400F1EB%40chrismccord.com.




-- 

Regards,
Bruce Tate
CEO



Aaron Ross
unread,
Dec 16, 2021, 7:17:30 PM12/16/21
to elixir-lang-core
What about rebinding within a for being more explicit? None of for comprehensions’ existing magic makes me uncomfortable because they’re all either entirely contained in the for declaration, or very explicit in the body (reduce/acc -> ... in the body feels sufficiently explicit). Instead of overloading =, since it already means “rebind this variable”, what if there were a directive like let!(var, value) or rebind!(var, value) or similar?


As mentioned previously, I think let initialization should be contained within the for declaration, I don’t think you should be able to reuse variables declared in the outer scope.


{sections, _} =
  for section <- sections,
      let: {section_counter = 1, lesson_counter = 1} do
    if section["reset_lesson_position"], do: let!(lesson_counter, 1)

    {lessons, lesson_counter} =
      for lesson <- section["lessons"], let: lesson_counter do
        lesson = Map.put(lesson, "position", lesson_counter)
        let!(lesson_counter, &(& + 1)) # could also accept a 1-arity function as the second argument,
                                       # similar to `Agent.update/2` and friends
        lesson
      end

    section =
      section
      |> Map.put("lessons", lessons)
      |> Map.put("position", section_counter)

    let!(section_counter, &(& + 1))
    section
  end

​
Stefan Chrobot
unread,
Dec 16, 2021, 7:21:07 PM12/16/21
to elixir-l...@googlegroups.com
> let: section_counter <- 1, let: lesson_counter <- 1

My concern about this is that `<-` in for means extracting something from the collection, so giving it another meaning inside an option can be quite confusing.

Makes sense. If I'm not mistaken it actually means pulling the next item from an enumerable. How about trying to think of the for/let as generating a new enumerable (stream?) as the "loop" runs? So how about:

for section <- sections,
    section_counter <- let(1) do

end

Then how about "previous" instead of "let"?

for section <- sections,
    {_value, section_counter, lesson_counter} <- previous({nil, 1, 1}) do
  
  # ...
  {section, section_counter + 1, lesson_counter + 1}
end

The semantics would be that "<- previous" extracts the previous result of the do block. Once could as well do:

for n <- 1..5, p <- previous(1), do: p + n

Best,
Stefan


Sabiwara Yukichi
unread,
Dec 16, 2021, 8:25:26 PM12/16/21
to elixir-l...@googlegroups.com
Hi! I think this is a great idea, after having an initial knee-jerk reaction
 against it I realized this is actually removing a lot of boilerplate 
and helps close the gap with imperative languages for some types of 
algorithms.
But indeed, it seems like a potential source of confusion between how regular variables and comprehension variables behave.

Maybe
 a possibility could be to distinguish comprehension variables, for 
example by prefixing them in the same way as module attributes are 
prefixed with `@`. 
This could also solve the issue of keyword lists having plain atom keys and not carrying context.

{sections, _} =
  for section <- sections,
      let: [$section_counter: 1, $lesson_counter: 1] do
    $lesson_counter = if section["reset_lesson_position"], do: 1, else: $lesson_counter

    {lessons, $lesson_counter} =
      for lesson <- section["lessons"], let: $lesson_counter do
        lesson = Map.put(lesson, "position", $lesson_counter)
        $lesson_counter = $lesson_counter + 1
        lesson
      end

    section =
      section
      |> Map.put("lessons", lessons)
      |> Map.put("position", $section_counter)

    $section_counter = $section_counter + 1
    section
  end

We could maybe even remove the `let` keyword altogether?

for section <- sections, $section_counter = 1, $lesson_counter = 1 do

(and it this case would be returning a tuple of 3 for instance).
There
 are probably a lot of downsides I don't see, and it introduces a change
 in the language syntax, which might not be justified by this use case.

Cheers



-- 

You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4LmO8v%2BdzAtvNd8Ni%2BmWFgL0R%2B%3Dz-sVntuyp0ow%2BzaH4g%40mail.gmail.com.




Christopher Keele
unread,
Dec 17, 2021, 1:01:53 AM12/17/21
to elixir-lang-core
I love the thought put into this proposal, and really like the problem it is tackling! I am looking forward to the next proposal and will try to get to my inbox earlier for it.

Proposal Feedback

I mostly second the impressions voiced here, but really want to call attention to the criticism:

> this breaks refactoring for the inner contents of `for`

This is the real true deal-breaker for me. Referential transparency is a huge part of my mental model of Elixir and the key reason why it is such a joy to maintain code in. I am not sure if it is possible to introduce an imperative-loop construct that doesn't violate this property, so I may have to get over that. I do remember how painful it was to remove assignment-inside-ifs, though.

Replies

Re: for section <- sections, $section_counter = 1, $lesson_counter = 1 do

> Maybe a possibility could be to distinguish comprehension variables, for example by prefixing them in the same way as module attributes are prefixed with `@`.

This does elegantly solve my refactoring concern; in that "imperative" comprehension variables copied out of the comprehension could immediately raise a syntax error, as would moving them into a different comprehension that does not have them declared as imperative in the comprehension head. The compiler would also have to enforce never letting you use the same name with an imperative variables as with a normal one, to completely eliminate edge cases. I think this solution even works for nested comprehensions, though I still am not sure how that would work with the existing proposal.

> We could maybe even remove the `let` keyword altogether?

That  makes me really like syntax. We are not exactly running short on propositions but it nice to keep that overhead low. Also, the only other existing identifier syntax (module attributes) use a prefix/sigil approach as well, and this feels in the same category to me: we are introducing a different type of identifier with different scoping rules (even though what happens at compile time to it is wildly different).

Re: overloading the <- operator

> My concern about this is that `<-` in for means extracting something from the collection, so giving it another meaning inside an option can be quite confusing.

> If I'm not mistaken it actually means pulling the next item from an enumerable.

FWIW I've been writing Elixir for years and I still forget when I crack open a for or for a with that I need to be using <- . I've just internalized it as the "powerful SpecialForms clause operator". So I don't think allowing its use in other powerful new constructs, potentially nested in for or with, or inside their options lists, would be confusing, from my perspective at least.



José Valim
unread,
Dec 17, 2021, 2:14:13 AM12/17/21
to elixir-lang-core
Re: for section <- sections, $section_counter = 1, $lesson_counter = 1 do

I did consider introducing (precisely) $ for variables but my concern is that, by introducing special syntax, I believe most would expect it to be fully mutable, so you can modify it from any scope. That's why I decided to go with plain variables, because they already have a limited scope in Elixir and clear rules (but at the same time I agree that adding :let would make those clear rules precisely more confusing!).



-- 

You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.


Sabiwara Yukichi
unread,
Dec 17, 2021, 5:40:14 AM12/17/21
to elixir-l...@googlegroups.com
Indeed this doesn't address the issue of the level of nesting, and is confusing in this case.

The new syntax could maybe include the level of nesting information somehow, e,g. ` $section_counter` in the parent loop, ` $$section_counter` in the child loop...?
Or $1.section_counter = 1 (parent),  $2.section_counter = 1 (child)? (slightly inspired by &1).

Another
 way to deal with this inconsistency could be to forbid nested 
comprehension with variables, and require to extract as a new function 
(in the same way the & cannot be nested and require to use fn).
Most examples would probably be easier to understand this way anyway, but this might limit the power of the feature.

Or
 maybe just having the compiler raising an error if trying to re-assign 
within a nested block, with a helpful beginner-friendly message, could be enough to clear 
this confusion?
I think this is not so much harder to figure than the fact than a re-assignment within an if doesn't work as in imperative languages.

By looking at the examples here, I feel that the last one might be the most elegant of these 3 ideas: https://gist.github.com/sabiwara/97c480c2076666ba9b98cf7a142a5a0f



To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4%2BsbvBxoj1mECXzBna%3DJE-R8%2Bj-CBuRZvgAf%2BsLp2aMjw%40mail.gmail.com.




Christopher Keele
unread,
Dec 17, 2021, 7:32:18 AM12/17/21
to elixir-lang-core
> I did consider introducing (precisely) $ for variables but my concern is that, by introducing special syntax, I believe most would expect it to be fully mutable, so you can modify it from any scope.

I am not sure if I can envision a way to allow imperative-ish variables without introducing special semantics. So I feel like supporting the new semantics with special syntax would allow us to set correct expectations about its scope and mutability when introducing/documenting it!

In fact, making special-semantics different syntactically to be more googleable is a perk over plain variables in my mind. For example, searching "ruby double at" (a comparatively oblique ruby language identifier feature, @@class_variables), returns an appropriate top result (from an incognito browser session, no less)! So maybe an "elixir dollar variable"  google search is a reasonable standard to hold ourselves to.

João Pedro Evangelista
unread,
Dec 17, 2021, 10:20:39 AM12/17/21
to elixir-lang-core
I was also skeptical at first, an thought as another construct to newcomers to learn, but on a perspective that it would make it easy to manage state while iterating it becomes more useful, it is true that can be error prone to have that many state wrangling while doing nested traversal. The name  "let" still does not sit well with me because it is not present in any other part of the language,  but we do have "state", "acc", "accumulator" for example, which could translate better when going to the already present Enum module functions and recursion.
João Pedro Evangelista
unread,
Dec 17, 2021, 10:35:15 AM12/17/21
to elixir-lang-core

> In fact, making special-semantics different syntactically to be more googleable 

Also more easily scannable while reading the code, we would know that this variable has more meaning among the other ones
On Friday, December 17, 2021 at 9:32:18 AM UTC-3 christ...@gmail.com wrote:
Zach Daniel
unread,
Dec 17, 2021, 12:59:19 PM12/17/21
to elixir-l...@googlegroups.com
I may not fully be understanding what we're looking for here, but it seems like this would effectively be the equivalent of:

```
result = 
for foo ← [1, 2, 3], reduce: %{acc: [], counter: 0} do
  %{acc: acc, counter: counter} ->
    new_acc = some_calc(acc)
    %{acc: new_acc, counter: counter + 1}
end

actual_result = result.acc
```

I'm wondering if we could just introduce an additional `state` that you match on, and return as a tuple from the for loop body? I think this is similar to what the original proposal wanted, but it involves only knowing that if you use the `state` option, you need to return a tuple of the for loop result and the new state. And it looks similar to a genserver in that regard, which makes it feel reasonably conventional.

```
result = 
  for foo ← [1, 2, 3], reduce: [], state: %{counter: 0} do
    acc, state →
     {some_calc(acc, state.counter), %{state | counter: state.counter + 1}}
  end
```

 

Sent via Superhuman



To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/10d18915-55da-4b52-8e12-0992625039e3n%40googlegroups.com.

Zach Daniel
unread,
Dec 17, 2021, 1:01:03 PM12/17/21
to elixir-l...@googlegroups.com
Oh, and if you don't have `reduce` then it just becomes the first thing, e.g

```
for foo ← [1, 2, 3], state: %{counter: 0} do
  state →
    {some_calc(acc, state.counter), %{state | counter: state.counter + 1}}
end
```

Sent via Superhuman


On Fri, Dec 17, 2021 at 12:59 PM, Zach Daniel <zachary....@gmail.com> wrote:
I may not fully be understanding what we're looking for here, but it seems like this would effectively be the equivalent of:

```
result = 
for foo ← [1, 2, 3], reduce: %{acc: [], counter: 0} do
  %{acc: acc, counter: counter} ->
    new_acc = some_calc(acc)
    %{acc: new_acc, counter: counter + 1}
end

actual_result = result.acc
```

I'm wondering if we could just introduce an additional `state` that you match on, and return as a tuple from the for loop body? I think this is similar to what the original proposal wanted, but it involves only knowing that if you use the `state` option, you need to return a tuple of the for loop result and the new state. And it looks similar to a genserver in that regard, which makes it feel reasonably conventional.

```
result = 
  for foo ← [1, 2, 3], reduce: [], state: %{counter: 0} do
    acc, state →
     {some_calc(acc, state.counter), %{state | counter: state.counter + 1}}
  end
```



Sent via Superhuman

 
José Valim
unread,
Dec 17, 2021, 1:04:33 PM12/17/21
to elixir-lang-core
Hi Zach,

I recommend trying out your approach on the original problem. You will see that you will need to accumulate elements, add Enum.reverse/1, etc. All which will make the solution noisier.






-- 

You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/kxaott2e.e380bedb-a7bd-4f10-8d2c-573d453c6880%40we.are.superhuman.com.




Zach Daniel
unread,
Dec 17, 2021, 1:19:37 PM12/17/21
to elixir-l...@googlegroups.com
The first approach I described was just me confirming my understanding of a corollary of the desired behavior that we can do now. The original problem stated using the method I showed in my second example would look like this, which seems pretty clean to me.

```
section_counter = 1
lesson_counter = 1

{sections, _} =
  for section <- sections,
      state: %{section_counter: section_counter, lesson_counter: lesson_counter} do
    %{lesson_counter: lesson_counter, section_counter: section_counter} ->
      lesson_counter = if section["reset_lesson_position"], do: 1, else: lesson_counter

      {lessons, lesson_counter} =
        for lesson <- section["lessons"], state: lesson_counter do
          lesson_counter ->
            lesson = Map.put(lesson, "position", lesson_counter)
            {lesson, lesson_counter + 1}
        end

      section =
        section
        |> Map.put("lessons", lessons)
        |> Map.put("position", section_counter)

      {section, %{section_counter: section_counter + 1, lesson_counter: lesson_counter}}
  end
```
Although I did originally forget to include the fact that if `state` is specified then not only do you match on it and return a tuple, but that the comprehension would return a tuple as well.
 

Sent via Superhuman


On Fri, Dec 17, 2021 at 1:04 PM, José Valim <jose....@dashbit.co> wrote:
Hi Zach,

I recommend trying out your approach on the original problem. You will see that you will need to accumulate elements, add Enum.reverse/1, etc. All which will make the solution noisier.

On Fri, Dec 17, 2021 at 6:59 PM Zach Daniel <zachary.s.daniel@gmail.com> wrote:
I may not fully be understanding what we're looking for here, but it seems like this would effectively be the equivalent of:

```
result = 
for foo ← [1, 2, 3], reduce: %{acc: [], counter: 0} do
  %{acc: acc, counter: counter} ->
    new_acc = some_calc(acc)
    %{acc: new_acc, counter: counter + 1}
end

actual_result = result.acc
```

I'm wondering if we could just introduce an additional `state` that you match on, and return as a tuple from the for loop body? I think this is similar to what the original proposal wanted, but it involves only knowing that if you use the `state` option, you need to return a tuple of the for loop result and the new state. And it looks similar to a genserver in that regard, which makes it feel reasonably conventional.

```
result = 
  for foo ← [1, 2, 3], reduce: [], state: %{counter: 0} do
    acc, state →
     {some_calc(acc, state.counter), %{state | counter: state.counter + 1}}
  end
```


Sent via Superhuman



To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.





-- 

You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.





-- 

You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4KYBajtuG0sAA%2BOtaWDo8AKn5csz6irK3f-Y1BLmrqvbw%40mail.gmail.com.

José Valim
unread,
Dec 17, 2021, 1:45:50 PM12/17/21
to elixir-l...@googlegroups.com
That’s what I meant. The reduce solution requires the reverse. The state proposal is very close to the Enum.map_reduce/3 solution, as it keeps all of its noise in passing the accumulator, matching on it, and returning it.

I am sure it is clear for an Elixir developer but so is Enum.map_reduce/3. Can we find a point midway? Maybe yes, maybe no. :)

PS: And, although we got used with for+:reduce, we have to admit that the fact the accumulator is matched in a clause inside do-end is a specific behavior to for that we partly accept because we got used to it!






-- 

You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.








-- 

You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.





-- 

You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.


José Valim
unread,
Dec 17, 2021, 1:52:33 PM12/17/21
to elixir-l...@googlegroups.com
Apologies, I re-read my first reply to you and it was definitely not clear I was taking about both approaches.
Zach Daniel
unread,
Dec 17, 2021, 1:55:00 PM12/17/21
to elixir-l...@googlegroups.com
Very true on the match inside `do/end`. Maybe controversial, but another potential idea would be to reduce the noise in the matches by supporting the js style object destructuring of `%{foo, bar}`. I can see how that is just a bit of a bandaid on top of the problem. I definitely see the problem statement here, but the proposal of the automatically tracked variables just seems difficult to swallow. It adds a whole new class of problem when working inside of loop bodies. If it would be added, I personally feel like a new designator like `$var` would actually be in order, so that someone doesn't refactor something without realizing they are actually affecting all other iterations. I'm definitely not in the camp of "we can do this with functions now, so lets not change anything". I often feel like comprehensions are just like 80% of the way to being a god-mode tool, so I'm all in favor of improving this aspect of it. I almost always choose functions  + composition at the moment just because I know I won't end up limited/having to refactor it to the same degree as I might w/ a for comprehension now.
 

Sent via Superhuman


On Fri, Dec 17, 2021 at 1:45 PM, José Valim <jose....@dashbit.co> wrote:
That’s what I meant. The reduce solution requires the reverse. The state proposal is very close to the Enum.map_reduce/3 solution, as it keeps all of its noise in passing the accumulator, matching on it, and returning it.

I am sure it is clear for an Elixir developer but so is Enum.map_reduce/3. Can we find a point midway? Maybe yes, maybe no. :)

PS: And, although we got used with for+:reduce, we have to admit that the fact the accumulator is matched in a clause inside do-end is a specific behavior to for that we partly accept because we got used to it!
On Fri, Dec 17, 2021 at 19:19 Zach Daniel <zachary.s.daniel@gmail.com> wrote:
The first approach I described was just me confirming my understanding of a corollary of the desired behavior that we can do now. The original problem stated using the method I showed in my second example would look like this, which seems pretty clean to me.

```
section_counter = 1
lesson_counter = 1

{sections, _} =
  for section <- sections,
      state: %{section_counter: section_counter, lesson_counter: lesson_counter} do
    %{lesson_counter: lesson_counter, section_counter: section_counter} ->
      lesson_counter = if section["reset_lesson_position"], do: 1, else: lesson_counter

      {lessons, lesson_counter} =
        for lesson <- section["lessons"], state: lesson_counter do
          lesson_counter ->
            lesson = Map.put(lesson, "position", lesson_counter)
            {lesson, lesson_counter + 1}
        end

      section =
        section
        |> Map.put("lessons", lessons)
        |> Map.put("position", section_counter)

      {section, %{section_counter: section_counter + 1, lesson_counter: lesson_counter}}
  end
```
Although I did originally forget to include the fact that if `state` is specified then not only do you match on it and return a tuple, but that the comprehension would return a tuple as well.

Sent via Superhuman


On Fri, Dec 17, 2021 at 1:04 PM, José Valim <jose.valim@dashbit.co> wrote:
Hi Zach,

I recommend trying out your approach on the original problem. You will see that you will need to accumulate elements, add Enum.reverse/1, etc. All which will make the solution noisier.


To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.





-- 

You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.








-- 

You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.





-- 

You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.





-- 

You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4KYCQC%2BYx8ibOW4Gbnbj7161BvvkbDSdE9oaV_zkUYhLw%40mail.gmail.com.

Reply all
Reply to author
Forward
0 new messages