Proposal: Add optional `into:` argument to Enum.map

76 views
Skip to first unread message

Wiebe-Marten Wijnja

unread,
Jul 7, 2016, 1:23:15 PM7/7/16
to elixir-lang-core
I propose to add `into: collectable` as optional third argument to Enum.map. 

This would allow mapping over things without first converting it to a list and then convert it back.

So:

%{"a" => 1, "b" => 2}
|> Enum.map(fn {k, v} -> {String.upcase(k), v} end)
|> Enum.into(%{})

could be written as:

%{"a" => 1, "b" => 2}
|> Enum.map(fn {k, v} -> {String.upcase(k), v} end, into: %{})

Not having to do the list conversion in the middle might also improve performance.


Right now, enumerating and collecting in one go is of course possible using `Kernel.SpecialForms.for`. `for` is however not very pipeline-friendly.


What do you think?


Sincerely,

~Wiebe-Marten

Andrea Leopardi

unread,
Jul 7, 2016, 1:24:16 PM7/7/16
to elixir-l...@googlegroups.com
Enum.into/3 should do what you want :)
--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/23c345b9-9ba1-48ea-8db0-6d2e9e700473%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


--

Andrea Leopardi

Peter Hamilton

unread,
Jul 7, 2016, 1:35:45 PM7/7/16
to elixir-l...@googlegroups.com

Does the Enumerable protocol enable the performance improvement of not converting to a list? I don't think it does.


To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAM9Rf%2BLY6tyh%3DE6iM%2BA7rmyY8dKP-ddCJpB%2BWZDn0jCuJ7FKaA%40mail.gmail.com.

Michał Muskała

unread,
Jul 7, 2016, 2:48:39 PM7/7/16
to elixir-l...@googlegroups.com
What do you mean by not converting to a list?
The fastest way right now of doing a map on a map is actually to convert to list, map and convert back to a map.
I’d suspect that’s the case with a lot of data structures as well, actually. Lists are simply really good for recursion.
The benefit is even greater if you do multiple operations on those lists, as the conversion happens (ideally) only
once.

Michał.
signature.asc

Ben Wilson

unread,
Jul 7, 2016, 6:23:36 PM7/7/16
to elixir-lang-core
This is correct. map |> Enum.map(&fun) |> Map.new is the fastest possible way to take a map and iterate over all of its keys.

Reason being, building a new list of {k, v} pairs and calling the `:maps.from_list` NIF always beats out N Map.put calls, which is how into: %{} works. It also creates less garbage. Map.new is basically just :maps.from_list if it gets passed a list.
Michał.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.


--

Andrea Leopardi


--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAM9Rf%2BLY6tyh%3DE6iM%2BA7rmyY8dKP-ddCJpB%2BWZDn0jCuJ7FKaA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Peter Hamilton

unread,
Jul 7, 2016, 6:25:28 PM7/7/16
to elixir-lang-core
To be fair, :maps.map is the fastest. But it's about as fast as :lists.map . Enum has a lot of overhead.

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/23c345b9-9ba1-48ea-8db0-6d2e9e700473%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


--

Andrea Leopardi


--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/af052020-b329-49e0-a7b7-f0bf487c123c%40googlegroups.com.

Ben Wilson

unread,
Jul 7, 2016, 6:28:58 PM7/7/16
to elixir-lang-core
Not for lists it doesn't https://github.com/elixir-lang/elixir/blob/cbe03987daa9c01a9d7ce99137a1d76535efb5f9/lib/elixir/lib/enum.ex#L1185

Enum.map is for all intents and purposes exactly as fast as :lists.map. :maps.fold just converts a map to a list and folds over it, and then turns it back into a map.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.


--

Peter Hamilton

unread,
Jul 7, 2016, 6:41:14 PM7/7/16
to elixir-lang-core
My benchmarks were on 1.2.4. Map.new got much faster due to https://github.com/elixir-lang/elixir/commit/11f046a7f5a8a69b81b9627de0312e435c8e6b60 , by none other than Ben Wilson.

Thanks for that!

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/23c345b9-9ba1-48ea-8db0-6d2e9e700473%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


--

Andrea Leopardi


--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/1afb754e-5d9b-41ff-a855-161610e63908%40googlegroups.com.

José Valim

unread,
Jul 7, 2016, 6:41:15 PM7/7/16
to elixir-l...@googlegroups.com
To be fair, :maps.map is the fastest. But it's about as fast as :lists.map . Enum has a lot of overhead.

Enum has very little overhead for built-in types as most of the implementations for built-in types are inlined.

Wiebe-Marten Wijnja

unread,
Jul 8, 2016, 1:48:03 AM7/8/16
to elixir-lang-core
You are right! I completely missed that there was a ternary version of `Enum.into`.

Let me refine the proposal:

- Update the documentation of `Enum.map` to reference `Enum.into/3`.
- Increase performance of `Enum.into/3` when the collectable is a map, by collecting to a list and in the end call `:maps.from_list` on the result. (As Ben Wilson explained, this is a lot faster/cleaner than n `Map.put` calls)


~Wiebe-Marten


On Thursday, July 7, 2016 at 7:24:16 PM UTC+2, Andrea Leopardi wrote:
Enum.into/3 should do what you want :)

On Thursday, 7 July 2016, Wiebe-Marten Wijnja <w.m.w...@panache-it.com> wrote:
I propose to add `into: collectable` as optional third argument to Enum.map. 

This would allow mapping over things without first converting it to a list and then convert it back.

So:

%{"a" => 1, "b" => 2}
|> Enum.map(fn {k, v} -> {String.upcase(k), v} end)
|> Enum.into(%{})

could be written as:

%{"a" => 1, "b" => 2}
|> Enum.map(fn {k, v} -> {String.upcase(k), v} end, into: %{})

Not having to do the list conversion in the middle might also improve performance.


Right now, enumerating and collecting in one go is of course possible using `Kernel.SpecialForms.for`. `for` is however not very pipeline-friendly.


What do you think?


Sincerely,

~Wiebe-Marten

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.


--

Andrea Leopardi

Ben Wilson

unread,
Jul 8, 2016, 7:44:11 AM7/8/16
to elixir-lang-core
I looked at having a fast path for Enum.into when I did the original Map.new upgrades and we decided against it. When the goal is to create a map from scratch |> Map.new is the clearest way to write that. The fast path for Enum.into would require that the second arg is BOTH a map AND an empty map, otherwise you can't use Map.new. If Enum.into(%{}) were the best way to create a new map from some other thing it would be worth the extra code path but I see Map.new as superior for that purpose.

I do agree that documentation in Enum could mention that. I'm inclined to stick it in the moduledoc though because it applies to conceptually to the whole module. By way of example, it's just as relevant to filter as it is to map.

Aleksei Magusev

unread,
Jul 9, 2016, 2:35:56 PM7/9/16
to elixir-lang-core
From now on we're rewriting Enum.into/2,3 with empty map into Map.new/1,2 during compilation. :)

Wiebe-Marten Wijnja

unread,
Jul 12, 2016, 4:13:50 PM7/12/16
to elixir-lang-core
Awesome! :D
Reply all
Reply to author
Forward
0 new messages