[erlang-questions] Unexpected limitations

Stu Bailey

unread,

Mar 27, 2013, 1:19:24 AM3/27/13

to Erlang/OTP discussions

Hi all,

I have an 8 core MacBook Pro Retina Display with 16GB of memory. I am running R16B with no configure options...but I did install libatomic_ops-7.2 . I tried a very simple list allocation test which surprisingly failed with a list size of 100 million. I first made a list with 10 million integers. The beam hung on 100 million. Thinking it was something with lists:seq, I just wrote a dumb test to build big lists of single character atoms. That also failed at 100 million. Is there a known limitation to the size of lists in Erlang?

bash-3.2$ uname -a

Darwin myhost.local 12.3.0 Darwin Kernel Version 12.3.0: Sun Jan 6 22:37:10 PST 2013; root:xnu-2050.22.13~1/RELEASE_X86_64 x86_64

bash-3.2$ erl

Erlang R16B (erts-5.10.1) [source] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false]

My Erlang Environment

Eshell V5.10.1 (abort with ^G)

1> lists:seq(1,10000000).

[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,

23,24,25,26,27,28,29|...]

2> lists:seq(1,100000000).

beam.smp(85491,0xb039d000) malloc: *** mmap(size=458227712) failed (error code=12)

*** error: can't allocate region

*** set a breakpoint in malloc_error_break to debug

beam.smp(85491,0xb039d000) malloc: *** mmap(size=659554304) failed (error code=12)

*** error: can't allocate region

*** set a breakpoint in malloc_error_break to debug

beam.smp(85491,0xb039d000) malloc: *** mmap(size=791674880) failed (error code=12)

*** error: can't allocate region

*** set a breakpoint in malloc_error_break to debug

beam.smp(85491,0xb039d000) malloc: *** mmap(size=791674880) failed (error code=12)

*** error: can't allocate region

*** set a breakpoint in malloc_error_break to debug

beam.smp(85491,0xb039d000) malloc: *** mmap(size=791674880) failed (error code=12)

*** error: can't allocate region

*** set a breakpoint in malloc_error_break to debug

beam.smp(85491,0xb039d000) malloc: *** mmap(size=790962176) failed (error code=12)

*** error: can't allocate region

*** set a breakpoint in malloc_error_break to debug

beam.smp(85491,0xb039d000) malloc: *** mmap(size=790962176) failed (error code=12)

*** error: can't allocate region

*** set a breakpoint in malloc_error_break to debug

Killed: 9

================

-module(big_lists).

-compile([export_all]).

make_list(0,Acc)->

Acc;

make_list(Size,Acc) ->

make_list(Size-1, [a] ++ Acc).

===================

5> c(big_lists).

{ok,big_lists}

6> big_lists:make_list(2,[]).

[a,a]

7> big_lists:make_list(100000,[]).

[a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a|...]

8> big_lists:make_list(1000000,[]).

[a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a|...]

9> big_lists:make_list(10000000,[]).

[a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a|...]

10> big_lists:make_list(100000000,[]).

beam.smp(85534,0xb039d000) malloc: *** mmap(size=381681664) failed (error code=12)

*** error: can't allocate region

*** set a breakpoint in malloc_error_break to debug

beam.smp(85534,0xb039d000) malloc: *** mmap(size=659554304) failed (error code=12)

*** error: can't allocate region

*** set a breakpoint in malloc_error_break to debug

beam.smp(85534,0xb039d000) malloc: *** mmap(size=791674880) failed (error code=12)

*** error: can't allocate region

*** set a breakpoint in malloc_error_break to debug

[a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a|...]

beam.smp(85534,0xb039d000) malloc: *** mmap(size=950009856) failed (error code=12)

*** error: can't allocate region

*** set a breakpoint in malloc_error_break to debug

beam.smp(85534,0xb039d000) malloc: *** mmap(size=950009856) failed (error code=12)

*** error: can't allocate region

*** set a breakpoint in malloc_error_break to debug

beam.smp(85534,0xb039d000) malloc: *** mmap(size=949153792) failed (error code=12)

*** error: can't allocate region

*** set a breakpoint in malloc_error_break to debug

beam.smp(85534,0xb039d000) malloc: *** mmap(size=949153792) failed (error code=12)

*** error: can't allocate region

*** set a breakpoint in malloc_error_break to debug

Crash dump was written to: erl_crash.dump

eheap_alloc: Cannot allocate 949152844 bytes of memory (of type "heap").

Abort trap: 6

Bengt Kleberg

unread,

Mar 27, 2013, 2:21:56 AM3/27/13

to Erlang/OTP discussions

Greetings,

Is your Erlang/OTP 32 or 64 bit?

bengt

> _______________________________________________
> erlang-questions mailing list
> erlang-q...@erlang.org
> http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
erlang-q...@erlang.org
http://erlang.org/mailman/listinfo/erlang-questions

Danil Zagoskin

unread,

Mar 27, 2013, 2:31:39 AM3/27/13

to Stu Bailey, Erlang/OTP discussions

Hi, Stu.

Does your beam crash when you call lists:duplicate(100000000, a) ?

The difference is lists:duplicate is tail-recursive while lists:seq and big_lists:make_list are body-recursive so you can hit some stack size limit, not lists length one.

2013/3/27 Stu Bailey <stu.b...@gmail.com>

Lukas Larsson

unread,

Mar 27, 2013, 3:37:25 AM3/27/13

to Danil Zagoskin, Erlang/OTP discussions

Hello Stu!

As Bengt pointed out you need a 64 bit erlang emulator because the list you created in 2 words * 100 million which would be about 1.6 GB. When the copying GC kicks in that expands to very close to 4GB which is the max in a 32 bit emulator. To compile a 64 but emulator on OS X see http://www.erlang.org/doc/installation_guide/INSTALL.html#id72320

Lukas

Stu Bailey

unread,

Mar 27, 2013, 10:24:20 AM3/27/13

to Erlang/OTP discussions

Everything works with 64bit emulator, sorry for the false alarm. Thanks!

Reply all

Reply to author

Forward