Note: This proposal contains images and rich text that may not display correctly in your email. If so, you can read this proposal in a gist.
There is prior art in languages like Common Lisp, Haskell,
and even in C# with LINQ on having very powerful comprehensions as part
of the language. While Elixir comprehensions are already very
expressive, allowing you to map
, filter
, reduce
, and collect
over multiple enumerables at the same time, it is still not capable of expressing other constructs, such as map_reduce
.
The challenge here is how to continue adding more
expressive power to comprehensions without making the API feel massive.
That's why, 7 years after v1.0, only two new options have been added to
comprehensions, :uniq
and :reduce
, to a total of 3 (:into
, :uniq
, and :reduce
).
I have been on the record a couple times saying that, while many problems are more cleanly solved with recursion, there is a category of problems that are much more elegant with imperative loops. One of those problems have been described in the "nested-data-structures-traversal" repository, with solutions available in many different languages. Please read the problem statement in said repository, as I will assume from now on that you are familiar with it.
Personally speaking, the most concise and clear solution is the Python one, which I reproduce here:
section_counter = 1 lesson_counter = 1 for section in sections: if section["reset_lesson_position"]: lesson_counter = 1 section["position"] = section_counter section_counter += 1 for lesson in section["lessons"]: lesson["position"] = lesson_counter lesson_counter += 1
There are many things that make this solution clear:
Let's compare it with the Elixir solution I wrote and personally prefer. I am pasting an image below which highlights certain aspects:
Lack of reassignment: in Elixir, we can't reassign variables, we can only rebind them. The difference is, when you do var = some_value
inside a if
, for
, etc, the value won't "leak" to the outer scope. This implies two things in the snippet above:
Enum.map_reduce/3
and pass the state in and out (highlighted in red)Lack of mutability: even though we set the lesson counter inside the inner map_reduce
, we still need to update the lesson inside the session (highlighted in green)
Lack of sensitive whitespace: we have two additional lines with end
in them (highlighted in blue)
As you can see, do-end blocks add very litte noise to the final solution compared to sensitive whitespace. In fact, the only reason I brought it up is so we can confidently discard it from the discussion from now on. And also because there is zero chance of the language suddenly becoming whitespace sensitive.
There is also zero chance of us introducing reassignment and making mutability first class in Elixir too. The reason for this is because we all agree that, the majority of the time, lack of reassignment and lack of mutability are features that make our code more readable and understandable in the long term. The snippet above is one of the few examples where we are on the wrong end of the trade-offs.
Therefore, how can we move forward?
Comprehensions in Elixir have always been a syntax sugar
to more complex data-structure traversals. Do you want to have the
cartesian product between all points in x
and y
? You could write this:
Enum.flat_map(x, fn i -> Enum.map(y, fn j -> {i, j} end) end)
Or with a comprehension:
for i <- x, j <- y, do: {i, j}
Or maybe you want to brute force your way into finding Pythagorean Triples?
Enum.flat_map(1..20, fn a -> Enum.flat_map(1..20, fn b -> 1..20 |> Enum.filter(fn c -> a*a + b*b == c*c end) |> Enum.map(fn c -> {a, b, c} end) end) end)
Or with a comprehension:
for a <- 1..20, b <- 1..20, c <- 1..20, a*a + b*b == c*c, do: {a, b, c}
There is no question the comprehensions are more concise and clearer, once you understand their basic syntax elements (which are, at this point, common to many languages).
As mentioned in the introduction, we can express map
, filter
, reduce
, and collect
inside comprehensions. But how can we represent map_reduce
in a clear and concise way?
:map_reduce
optionSince we have :reduce
in comprehensions, we could introduce :map_reduce
. The solution above would look like this:
{sections, _acc} = for section <- sections, map_reduce: {1, 1} do {section_counter, lesson_counter} -> lesson_counter = if section["reset_lesson_position"], do: 1, else: lesson_counter {lessons, lesson_counter} = for lesson <- section["lessons"], map_reduce: lesson_counter do lesson_counter -> {Map.put(lesson, "position", lesson_counter), lesson_counter + 1} end section = section |> Map.put("lessons", lessons) |> Map.put("position", section_counter) {section, {section_counter + 1, lesson_counter}} end
While there is a bit less noise compared to the original
solution, the reduction of noise mostly happened by the removal of
modules names and a few tokens, such as fn
, (
, and )
. In terms of implementation, there is still a lot of book keeping required to manage the variables. Can we do better?
:let
Our goal is to declare variables that are automatically
looped within the comprehension. So let's introduce a new option that
does exactly that: :let
. :let
expects one or a tuple of variables that will be reused across the comprehension. At the end, :let
returns a tuple with the comprehension elements and the let variables.
Here is how the solution would look like:
section_counter = 1 lesson_counter = 1 {sections, _} = for section <- sections, let: {section_counter, lesson_counter} do lesson_counter = if section["reset_lesson_position"], do: 1, else: lesson_counter {lessons, lesson_counter} = for lesson <- section["lessons"], let: lesson_counter do lesson = Map.put(lesson, "position", lesson_counter) lesson_counter = lesson_counter + 1 lesson end section = section |> Map.put("lessons", lessons) |> Map.put("position", section_counter) section_counter = section_counter + 1 section end
The :let
option automatically takes care of
passing the variables across the comprehension, considerably cutting
down the noise, without introducing any mutability into the language. At
the end, for+:let
returns the result of the comprehension plus the :let
variables wrapped in a tuple.
Here are some extensions to the proposal above. Not all of them might be available on the initial implementation.
You can also initialize the variables within let
for convenience:
{sections, _} = for section <- sections, let: {section_counter = 1, lesson_counter = 1} do
This should be available in the initial implementation.
:reduce
vs :let
With :let
, :reduce
becomes somewhat redundant. For example, Enum.group_by/2
could be written as:
for {k, v} <- Enum.reverse(list), reduce: %{} do acc -> Map.update(acc, k, [v], &[v | &1]) end
with :let
:
{_, acc} = for {k, v} <- Enum.reverse(list), let: acc = %{} do acc = Map.update(acc, k, [v], &[v | &1]) end
The difference, however, is that :let
returns the collection, while :reduce
does not. While the Elixir compiler could be smart enough to optimize away building the collection in the :let
case if we don't use it, we may want to keep both :let
and :reduce
options for clarity. If this is the case, I propose to align the syntaxes such that :reduce
uses the same semantics as :let
. The only difference is the return type:
for {k, v} <- Enum.reverse(list), reduce: acc = %{} do acc = Map.update(acc, k, [v], &[v | &1]) end
This can be done in a backwards compatible fashion.
after
When you look at our solution to the problem using let
, we had to introduce temporary variables in order to update our let variables:
{lessons, lesson_counter} = for lesson <- section["lessons"], let: lesson_counter do lesson = Map.put(lesson, "position", lesson_counter) lesson_counter = lesson_counter + 1 lesson end
One extension is to add after
to the comprehensions, which are computed after the result is returned:
{lessons, lesson_counter} = for lesson <- section["lessons"], let: lesson_counter do Map.put(lesson, "position", lesson_counter) after lesson_counter = lesson_counter + 1 end
This does not need to be part of the initial implementation.
Feedback on the proposal and extensions is welcome!
for lesson <- lessons, let: lesson_counter do
if lesson.active? do
lesson_counter = lesson_counter + 1
end
end
for lesson <- lessons, let: lesson_counter do
Enum.each(lesson.sessions, fn(_) ->
lesson_counter = lesson_counter + 1
end)
end
--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4JjyZ2EUcYm1TA747pP2pTgDAD_%3DW%2BM9mSizFHJXFfqnQ%40mail.gmail.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CABu8xFAMx3hTfFbu%2B%2BN9XXnG5sEJ4zTSXdugSa96-u9uEGq%2B5g%40mail.gmail.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAD3kWz99qSby_7PjBT%3D68ztzmX-2tq%2B-xV7RXL_ze4ZXsaFp7w%40mail.gmail.com.
--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/22703d4b-60cb-4b0e-83d2-4a122f9147afn%40googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4KOF759bonEQ-5zX3_ebcO7y8ySOqS3_oeDcMdc-6Dwgw%40mail.gmail.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CABu8xFCiT_egSe-PmUmoQR%2BmQDmTigvJfF_wKman9_FQKrqYuA%40mail.gmail.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CABu8xFCiT_egSe-PmUmoQR%2BmQDmTigvJfF_wKman9_FQKrqYuA%40mail.gmail.com.
The
:let
option automatically takes care of passing the variables across the comprehension, considerably cutting down the noise, without introducing any mutability into the language. At the end,for+:let
returns the result of the comprehension plus the:let
variables wrapped in a tuple.
Extensions
Here are some extensions to the proposal above. Not all of them might be available on the initial implementation.
Let initialization
You can also initialize the variables within
let
for convenience:{sections, _} = for section <- sections, let: {section_counter = 1, lesson_counter = 1} doThis should be available in the initial implementation.
:reduce
vs:let
With
:let
,:reduce
becomes somewhat redundant. For example,Enum.group_by/2
could be written as:for {k, v} <- Enum.reverse(list), reduce: %{} do acc -> Map.update(acc, k, [v], &[v | &1]) endwith
:let
:{_, acc} = for {k, v} <- Enum.reverse(list), let: acc = %{} do acc = Map.update(acc, k, [v], &[v | &1]) end
The difference, however, is that
:let
returns the collection, while:reduce
does not. While the Elixir compiler could be smart enough to optimize away building the collection in the:let
case if we don't use it, we may want to keep both:let
and:reduce
options for clarity. If this is the case, I propose to align the syntaxes such that:reduce
uses the same semantics as:let
. The only difference is the return type:
for {k, v} <- Enum.reverse(list), reduce: acc = %{} do acc = Map.update(acc, k, [v], &[v | &1]) endThis can be done in a backwards compatible fashion.
after
When you look at our solution to the problem using
let
, we had to introduce temporary variables in order to update our let variables:{lessons, lesson_counter} = for lesson <- section["lessons"], let: lesson_counter do lesson = Map.put(lesson, "position", lesson_counter) lesson_counter = lesson_counter + 1 lesson endOne extension is to add
after
to the comprehensions, which are computed after the result is returned:{lessons, lesson_counter} = for lesson <- section["lessons"], let: lesson_counter do Map.put(lesson, "position", lesson_counter) after lesson_counter = lesson_counter + 1 endThis does not need to be part of the initial implementation.
Summary
Feedback on the proposal and extensions is welcome!
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/b55240b6-5eb0-4cbc-a1cf-5016a130e1e0%40www.fastmail.com.
for {k, v} <- Enum.reverse(list), reduce: acc: %{} do acc = Map.update(acc, k, [v], &[v | &1]) end
Amos King, CEO
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/657C7140-D8C2-4FF6-8616-3CD92400F1EB%40chrismccord.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAJr6D4S3WfHgmryrz9o%3D1AFwebbLRfakUN6Vvi%3D5w8knYjVnPw%40mail.gmail.com.
for a <- 1..20, b <- 1..20, c <- 1..20, a*a + b*b == c*c
,
reduce: %{} do
acc -> Map.put(acc, {a, b}, c)
end
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/657C7140-D8C2-4FF6-8616-3CD92400F1EB%40chrismccord.com.
What about rebinding within a for
being more explicit? None of for
comprehensions’ existing magic makes me uncomfortable because they’re all either entirely contained in the for
declaration, or very explicit in the body (reduce/acc -> ...
in the body feels sufficiently explicit). Instead of overloading =
, since it already means “rebind this variable”, what if there were a directive like let!(var, value)
or rebind!(var, value)
or similar?
As mentioned previously, I think let
initialization should be contained within the for
declaration, I don’t think you should be able to reuse variables declared in the outer scope.
{sections, _} =
for section <- sections,
let: {section_counter = 1, lesson_counter = 1} do
if section["reset_lesson_position"], do: let!(lesson_counter, 1)
{lessons, lesson_counter} =
for lesson <- section["lessons"], let: lesson_counter do
lesson = Map.put(lesson, "position", lesson_counter)
let!(lesson_counter, &(& + 1)) # could also accept a 1-arity function as the second argument,
# similar to `Agent.update/2` and friends
lesson
end
section =
section
|> Map.put("lessons", lessons)
|> Map.put("position", section_counter)
let!(section_counter, &(& + 1))
section
end
> let: section_counter <- 1, let: lesson_counter <- 1My concern about this is that `<-` in for means extracting something from the collection, so giving it another meaning inside an option can be quite confusing.
--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4LmO8v%2BdzAtvNd8Ni%2BmWFgL0R%2B%3Dz-sVntuyp0ow%2BzaH4g%40mail.gmail.com.
--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/72ee4929-efde-476e-9124-bacd7460c486n%40googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4%2BsbvBxoj1mECXzBna%3DJE-R8%2Bj-CBuRZvgAf%2BsLp2aMjw%40mail.gmail.com.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/10d18915-55da-4b52-8e12-0992625039e3n%40googlegroups.com.
I may not fully be understanding what we're looking for here, but it seems like this would effectively be the equivalent of:```
result =for foo ← [1, 2, 3], reduce: %{acc: [], counter: 0} do
%{acc: acc, counter: counter} ->
new_acc = some_calc(acc)
%{acc: new_acc, counter: counter + 1}
end
actual_result = result.acc```I'm wondering if we could just introduce an additional `state` that you match on, and return as a tuple from the for loop body? I think this is similar to what the original proposal wanted, but it involves only knowing that if you use the `state` option, you need to return a tuple of the for loop result and the new state. And it looks similar to a genserver in that regard, which makes it feel reasonably conventional.```
result =for foo ← [1, 2, 3], reduce: [], state: %{counter: 0} doacc, state →
{some_calc(acc, state.counter), %{state | counter: state.counter + 1}}end```
Sent via Superhuman
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/10d18915-55da-4b52-8e12-0992625039e3n%40googlegroups.com.
--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/kxaott2e.e380bedb-a7bd-4f10-8d2c-573d453c6880%40we.are.superhuman.com.
Hi Zach,I recommend trying out your approach on the original problem. You will see that you will need to accumulate elements, add Enum.reverse/1, etc. All which will make the solution noisier.
On Fri, Dec 17, 2021 at 6:59 PM Zach Daniel <zachary.s.daniel@gmail.com> wrote:
I may not fully be understanding what we're looking for here, but it seems like this would effectively be the equivalent of:```
result =for foo ← [1, 2, 3], reduce: %{acc: [], counter: 0} do
%{acc: acc, counter: counter} ->
new_acc = some_calc(acc)
%{acc: new_acc, counter: counter + 1}
end
actual_result = result.acc```I'm wondering if we could just introduce an additional `state` that you match on, and return as a tuple from the for loop body? I think this is similar to what the original proposal wanted, but it involves only knowing that if you use the `state` option, you need to return a tuple of the for loop result and the new state. And it looks similar to a genserver in that regard, which makes it feel reasonably conventional.```
result =for foo ← [1, 2, 3], reduce: [], state: %{counter: 0} doacc, state →
{some_calc(acc, state.counter), %{state | counter: state.counter + 1}}end```
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/10d18915-55da-4b52-8e12-0992625039e3n%40googlegroups.com.
--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/kxaott2e.e380bedb-a7bd-4f10-8d2c-573d453c6880%40we.are.superhuman.com.
--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4KYBajtuG0sAA%2BOtaWDo8AKn5csz6irK3f-Y1BLmrqvbw%40mail.gmail.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/10d18915-55da-4b52-8e12-0992625039e3n%40googlegroups.com.
--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/kxaott2e.e380bedb-a7bd-4f10-8d2c-573d453c6880%40we.are.superhuman.com.
--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4KYBajtuG0sAA%2BOtaWDo8AKn5csz6irK3f-Y1BLmrqvbw%40mail.gmail.com.
--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/kxaprh1k.8bfd20dc-9ccf-46d7-b01a-9746ae11d488%40we.are.superhuman.com.
That’s what I meant. The reduce solution requires the reverse. The state proposal is very close to the Enum.map_reduce/3 solution, as it keeps all of its noise in passing the accumulator, matching on it, and returning it.I am sure it is clear for an Elixir developer but so is Enum.map_reduce/3. Can we find a point midway? Maybe yes, maybe no. :)PS: And, although we got used with for+:reduce, we have to admit that the fact the accumulator is matched in a clause inside do-end is a specific behavior to for that we partly accept because we got used to it!
On Fri, Dec 17, 2021 at 19:19 Zach Daniel <zachary.s.daniel@gmail.com> wrote:
The first approach I described was just me confirming my understanding of a corollary of the desired behavior that we can do now. The original problem stated using the method I showed in my second example would look like this, which seems pretty clean to me.section_counter = 1
```lesson_counter = 1{sections, _} =for section <- sections,state: %{section_counter: section_counter, lesson_counter: lesson_counter} do%{lesson_counter: lesson_counter, section_counter: section_counter} ->lesson_counter = if section["reset_lesson_position"], do: 1, else: lesson_counter{lessons, lesson_counter} =for lesson <- section["lessons"], state: lesson_counter dolesson_counter ->lesson = Map.put(lesson, "position", lesson_counter){lesson, lesson_counter + 1}endsection =section|> Map.put("lessons", lessons)|> Map.put("position", section_counter){section, %{section_counter: section_counter + 1, lesson_counter: lesson_counter}}end
```Although I did originally forget to include the fact that if `state` is specified then not only do you match on it and return a tuple, but that the comprehension would return a tuple as well.
On Fri, Dec 17, 2021 at 1:04 PM, José Valim <jose.valim@dashbit.co> wrote:
Hi Zach,I recommend trying out your approach on the original problem. You will see that you will need to accumulate elements, add Enum.reverse/1, etc. All which will make the solution noisier.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/10d18915-55da-4b52-8e12-0992625039e3n%40googlegroups.com.
--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/kxaott2e.e380bedb-a7bd-4f10-8d2c-573d453c6880%40we.are.superhuman.com.
--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4KYBajtuG0sAA%2BOtaWDo8AKn5csz6irK3f-Y1BLmrqvbw%40mail.gmail.com.
--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/kxaprh1k.8bfd20dc-9ccf-46d7-b01a-9746ae11d488%40we.are.superhuman.com.
--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4KYCQC%2BYx8ibOW4Gbnbj7161BvvkbDSdE9oaV_zkUYhLw%40mail.gmail.com.