[erlang-questions] Heap alloc error in lists:foldl/3

[Arif Ishaq]

Hi,

 

I'm a newbie, playing with a toy programme to see how parallel processing works in Erlang.

 

The goal is to calculate the value of pi using Leibnitz series. The function pi/2 takes as arguments the number of processes, Np, and the number of terms handled by each process, Nt.

 

The machine is an HP workstation with Windows Vista Enterprise.

 

 

Here's the code:

 

-module(pp).

-compile(export_all).

 

%% calculate PI using Np parallel processes, each calculating Nt terms of the

%% Liebnitz series

 

pi(Np, Nt) ->

    Collector = spawn(?MODULE, collect, [Np, self()]),

 

    [spawn_opt(

       fun() ->  Collector ! foldsum((N-1)*Nt, N*Nt -1) end      %% if I use sum, instead of foldsum, everything is ok

      ) || N <- lists:seq(1,Np)],

 

    receive

        Result ->

            io:format("result is ~p~n", [4 * Result])

    end.

 

 

sum(Kfrom, Kto) ->

    sum (Kfrom, Kto, 0).

 

sum(Kfrom, Kto, Sum) when Kfrom =:= Kto ->

    Sum;

sum(Kfrom, Kto, Sum) ->

    sum(Kfrom+1, Kto, Sum + contribution(Kfrom)).

 

 

foldsum(Kfrom, Kto) ->   

    lists:foldl(

      fun(K, Sum) -> Sum + contribution(K) end,

      0,

      lists:seq(Kfrom,Kto)).

 

 

contribution(K) ->

    Value = 1 / (2 * K + 1),

    case K rem 2 of

        0 ->

            Value;

        1 ->

            -Value

    end.

 

 

collect (Np, For) ->

    collect(Np, For, 0, 0).

 

collect (Np, For, Ncollected, Sum) when Np =:= Ncollected ->

    For ! Sum;

collect(Np, For, Ncollected, Sum) ->

    receive

        R ->

            collect(Np, For, Ncollected + 1, Sum + R)

    end.

 

 

 

If I try to calculate pi using 16 processes, each processing 1 million terms, the result is ok.

If, on the other hand, I try with 10 million terms each, erlang crashes.

Even after closing the werl window, the process keeps running and has to be killed brutally.

 

 

Erlang R15B (erts-5.9) [smp:4:4] [async-threads:0]

 

Eshell V5.9  (abort with ^G)

1> c(pp).

{ok,pp}

2> pp:pi(16,1000000).

result is 3.1415925910897737

ok

3> pp:pi(16,10000000).

 

Crash dump was written to: erl_crash.dump

eheap_alloc: Cannot allocate 40121760 bytes of memory (of type "heap").

 

 

Abnormal termination

 

If I try to read the crash dump with the crash dump viewer tool, I get an error saying the file is not an erlang crash dump:

erl_crash.dump is not an Erlang crash dump

 

 

Why is this happening? Shouldn't foldl be tail recursive? Is this a bug? Or am I doing something wrong?

 

 

Thanks and best regards

Arif Ishaq

 

 

[Ladislav Lenart]

Hello.

%% if I use sum, instead of foldsum, everything is ok

This is because foldsum/2 creates a list of 10 000 000 integers (call to
lists:seq/2). And you create Np of such lists in total, each one in a different
process. A list in Erlang is a chain of cons pairs (as in Lisp). Each cons
occupies 2 words, where one word is either 32bits or 64bits, depending on your
HW architecture. Thus one such list takes

10 000 000 * 8 = 80MB (on 32-bit architecture)

Hence it is fairly possible that your system simply ran out of memory.

The version using sum/2 does not exhibit this behaviour, because it runs in
constant space.


HTH,

Ladislav Lenart

> _______________________________________________
> erlang-questions mailing list
> erlang-q...@erlang.org
> http://erlang.org/mailman/listinfo/erlang-questions


_______________________________________________
erlang-questions mailing list
erlang-q...@erlang.org
http://erlang.org/mailman/listinfo/erlang-questions

[Arif Ishaq]

Thanks for your answer. I understand that 16 such lists would occupy at least 16 * 80 MB = 480 MB, but I have a machine with 2 GB of physical RAM and as much again of virtual memory. I was monitoring the memory usage in the system and it didn't exceed 1.5 GB.

I suppose that the problem was nevertheless just that. Is there any way of "catching" the eheap_alloc in order to terminate things gracefully?

Thanks again
Arif