lists:map vs list comprehensions

[sed]

It seems like these usually accomplish the same task. Is there an
advantage to one over the other?

[Garrett Smith]

IMO, it's strictly a matter of code readability. There are some cases
where one approach is obviously easier to read than another and you
should go with that.

I did profile these two methods out of curiosity, and it looks like
list comp is 2x faster -- but that's 2x faster than a very, very, very
small number so I wouldn't use that as a gauge.

[Jesse Gumm]

List comprehensions give flexibility at the cost of speed.  Comprehensions allow for pattern matching and guard expressions to leave out list members that aren't relevant.  Use lists:map if you want all the results, and use a comprehension if you need to filter the results in some custom way.

Example, assuming a list of fruits you have in quantities, in the format {fruit(atom),Qty(number)}

L = [{apple,5},{apple,6},{banana,7},{orange,3}].

Here are some things you can'd do with lists:map (you can do with other lists functions, but not map):

% List the quantities of apple records

% Will ignore any tuples that don't have the atom apple 

% as the first element in a 2-tuple.

AppleQtys = [Qty || {apple,Qty} <- L].

Doing the Same thing with the lists module would be something like:

Filter = fun({Fruit,Qty}) -> case Fruit of apple-> true;_ -> false end end,

ExtractQty = fun({_,Qty}) -> Qty end,

Filtered = lists:filter(Filter,L),

AppleQtys = lists:map(ExtractQty,Filtered).

Essentially, List comprehensions bring the flexibility of pattern matching to list:map.  In many cases, it can be a preference, but LC are much, much more flexible than lists:map.

You need to call a single function on every element of a list, and you want the return value of every one, use lists:map

You need to call a function only on certain members of a list, you probably want to use a list comprehension.

-Jesse

On Fri, Nov 12, 2010 at 12:51 PM, sed <sethae...@gmail.com> wrote:

It seems like these usually accomplish the same task. Is there an
advantage to one over the other?

--
Jesse Gumm
Sigma Star Systems
414.940.4866
gu...@sigma-star.com
http://www.sigma-star.com

[Martin Logan]

Comprehensions compile down to a form that is quite fast. They are comparable to any of the higher order functions these days. Comprehensions also can be used for side effects these days just like foreach - though it is an odd use to me. The compiler understands if you don't save the result. 

List comprehensions can be used like map, or like a fold where you only accumulate certain elements. It is really a matter of style. The way I break it down is this; if my filter and generators fit onto a single line easily I use a comprehension, if not, I use a higher order function because the fun syntax more easily accommodates multiple lines. Just my style though. 

Cheers,

Martin

[Jordan Wilberding]

Just to elaborate, comprehensions get compiled into recursion functions now. Used to be done with funs.

The side effects thing is the one major difference I know between comprehensions and map.

JW

[Jesse Gumm]

I did some speed test here, just out of curiousity, comparing map and LC with an extremely simple function, and here are my results:

Setup:

Time = fun(F) -> Start = now(),F(),timer:now_diff(now(),Start) end.

F = fun(X) -> X end.

Seq = lists:seq(1,1000000).

Results:

Using List Comprehension:

Time(fun() -> [F(X) || X<-Seq] end).

4435668

4488647

5084843

5444659

4528376

4482179

5242341

4431415

Using lists:map:

Time(fun() -> lists:map(F,Seq) end).

2024174

3576577

3327062

3321503

3993573

3698929

3333554

3332473

3314765

3336110

This is obviously a very simple case (a function that returns the value of its argument and called a million times), but it looks like lists:map, in that regard hovered around 3.5 seconds, while the semi-equivalent list comprehension hovered around 4.5 seconds.

Still fast either way.

[Garrett Smith]

Which version of Erlang?

[Jesse Gumm]

R13B04

[Martin Logan]

Jesse, there are a few things about your test that you would have to
change for an accurate result and a few extraneous things you could
get rid of in the computation. I am also guessing that perhaps you ran
this in the shell based on the very long times. I have created another
test that should handle what should be all the possible variables and
compiler optimizations so that we can get accurate results. Here it
is:

-module(test).

-export([run/0]).

run() ->

    Seq = lists:seq(1,1000000),
    comp(10, Seq),
    mapper(10, Seq).

mapper(0, _Seq) ->
    ok;
mapper(Iter, Seq) ->
    Now1 = now(),
    Res = lists:map(fun(N) -> N + 1 end, Seq),
    Now2 = now(),
    self() ! hd(Res),
    io:format("Map time: ~p~n", [timer:now_diff(Now2, Now1)]),
    mapper(Iter - 1, Seq).

comp(0, _Seq) ->
    ok;
comp(Iter, Seq) ->
    Now1 = now(),
    Res = [N + 1 || N <- Seq],
    Now2 = now(),
    self() ! hd(Res),
    io:format("Comprehension time: ~p~n", [timer:now_diff(Now2, Now1)]),
    comp(Iter - 1, Seq).

and here are the results

1> test:run().
Comprehension time: 96650
Comprehension time: 28795
Comprehension time: 30056
Comprehension time: 28711
Comprehension time: 29349
Comprehension time: 28781
Comprehension time: 29186
Comprehension time: 28917
Comprehension time: 30079
Comprehension time: 28814
Map time: 56983
Map time: 57545
Map time: 57082
Map time: 57151
Map time: 57469
Map time: 56867
Map time: 57706
Map time: 57467
Map time: 57088
Map time: 57701
ok

The comprehension in this case is about twice as fast as the higher
order function.

Cheers,
Martin

[Martin Logan]

One more interesting point:

-module(test).

-export([run/0]).

run() ->
Seq = lists:seq(1,1000000),
comp(10, Seq),
mapper(10, Seq).

mapper(0, _Seq) ->
ok;
mapper(Iter, Seq) ->
Now1 = now(),

lists:map(fun(N) -> N + 1 end, Seq),
Now2 = now(),

io:format("Map time: ~p~n", [timer:now_diff(Now2, Now1)]),
mapper(Iter - 1, Seq).

comp(0, _Seq) ->
ok;
comp(Iter, Seq) ->
Now1 = now(),

[N + 1 || N <- Seq],
Now2 = now(),

io:format("Comprehension time: ~p~n", [timer:now_diff(Now2, Now1)]),
comp(Iter - 1, Seq).

1> test:run().
Comprehension time: 10786
Comprehension time: 10869
Comprehension time: 10869
Comprehension time: 10849
Comprehension time: 10931
Comprehension time: 10952
Comprehension time: 10888
Comprehension time: 10857
Comprehension time: 11230
Comprehension time: 10967
Map time: 112854
Map time: 141781
Map time: 57468
Map time: 57838
Map time: 57732
Map time: 58423
Map time: 57160
Map time: 58190
Map time: 57134
Map time: 57663
ok

If I get rid of the assignment of the value for the list comprehension
and the actual use of that variable then I speed the time of the
comprehension up considerably. In fact, the use never even mattered,
the compiler seems only to check if there is an assignment of the
value of the list comprehension when deciding how to optimize.

Cheers,
Martin

[Jesse Gumm]

I hear ya. Taking the function call out of the comprehension would definitely speed things up, but how significantly so, i was not expecting.

My test was definitely not meant to be a rigorous and comprehensive comparison between map and comprehensions. Merely, I was going for a basic comparison for use-cases where someone newish to the language might choose between one or the other (like the OP's question about what's different between map and LC). Naturally, a function that only returns the passed value is a useless one. 

As you suspected, I did indeed run this in the shell (a bit of a distraction from work). And just kept pressing up-enter-up-enter-up-enter... Which certainly accounts for the wide range of times - your numbers are much more consistent.

And your revision, removing the assignment of the return list: that's clever of the compiler to know you don't care about the results. I had recently read about the compiler optimzations of LC and how it does little tricks like that for performance, like checking for side-effects or something.

As with anything, if subtle performance differences like these are important, you've got to benchmark it.

Thanks for the revisions, certainly more rigorous than my little whipped up turd of a test.

-Jesse

--
Jesse Gumm
Sigma Star Systems
414.940.4866

[Edward Browne]

Why the significant difference in time between the first list comp (96650) and the rest (~29000) in Martin's benchmark?

-Ed

R/
Edward Browne
brow...@gmail.com
936-689-8983

[Jordan Wilberding]

I'd guess caching.

JW

[Martin Logan]

I would say memory allocation - but the best way to see would be to
run a profile on it

[Jesse Gumm]

Interesting. For the sake of testing this further, I added a few more variations to this just to see the results.

List comprehension with an anonymous function (comp_fn),

List comprehension with a call to a function in the module in the comprehension (comp_local),

List comprehension with a call to a function in the module anonymized with fun adder/1 (comp_mfa),

Map with a call to function in the module, anonymized with fun adder/1 (mapper_mfa),

The results are interesting to me.

From fastest to slowest

comp(), % LC with no function call

comp_local() % LC with call to adder function

comp_fn() % LC with call to anonymous function,

mapper() % original mapper call with anonymous function

mapper_mfa() % mapper modified to fun adder/1 syntax to call the native function.

comp_mfa() % list comprehension with anonymized call to adder/1

The raw comprehension, with no function was still the fastest, and map, in general was slower.

The interesting thing, albeit sort of expected, was the time difference between the anonymous function call, and the local function call within the comprehension.  The surprising result for me particularly was that anonymizing the native call with "fun adder/1" seemingly made everything slower than just using a local anonymous function.

Since this is starting to get long, I'm gonna pastebin this:

Code: http://pastebin.com/ti5XUxYm

Results: http://pastebin.com/Wtz9Rhe4

-Jesse

[Jesse Gumm]

Thanks for the module, btw, Martin.  Made adding those variations and results easier than my "up-enter-repeat" method.