[Caml-list] memory usage

[Jean Krivine]

Dear list members,

I am trying to run a stochastic simulator (written in ocaml) on a huge
data set and I have the following error message:

sim(9595) malloc: *** mmap(size=1048576) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
Fatal error: out of memory.

My system:

Mac Pro running OS X 10.5.4
Processor: 2 x 2.8 GHz Quad-Core Intel Xeon
Memory: 10 GB 800 MHz DDR2 FB-DIMM

Does someone know what happened? Do you have any idea of any parameter
I could tune in order to avoid that?

Thank you very much!

Jean

_______________________________________________
Caml-list mailing list. Subscription management:
http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
Archives: http://caml.inria.fr
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
Bug reports: http://caml.inria.fr/bin/caml-bugs

[Till Varoquaux]

It is hard to tell without any more informations but sometimes the
garbage collector needs some gentle proding:

OCaml handles it's own memory but can be a bad citizen when it comes
to making room for others. Unfortunately ocaml also has a bit of a
double personality: it doesn't know much about resources used in
external libraries or even in some of its own library (e.g. on a 32
bits machine running out of addressable space because of
Bigarray.map_file is not unheard of).

If this is your problem, you can either sprinkle your source code with
calls to Gc.major or tweak it using Gc.set.

Till

--
http://till-varoquaux.blogspot.com/

[Richard Jones]

On Fri, Jul 11, 2008 at 03:49:26PM -0400, Jean Krivine wrote:
> I am trying to run a stochastic simulator (written in ocaml) on a huge
> data set and I have the following error message:

I can confirm that OCaml works fine with huge datasets, on 64 bit
platforms anyway.

> sim(9595) malloc: *** mmap(size=1048576) failed (error code=12)
> *** error: can't allocate region
> *** set a breakpoint in malloc_error_break to debug
> Fatal error: out of memory.
>
> My system:
>
> Mac Pro running OS X 10.5.4
> Processor: 2 x 2.8 GHz Quad-Core Intel Xeon
> Memory: 10 GB 800 MHz DDR2 FB-DIMM
>
> Does someone know what happened? Do you have any idea of any parameter
> I could tune in order to avoid that?

Is the compiler 32 bits or 64 bits on this machine? Try doing:

$ ocaml
# Sys.word_size ;;

It should print out either '32' or '64'.

Also run your program under whatever the OS X equivalent of 'strace'
is (ktrace?) to find out exactly why the mmap call fails.

OCaml <= 3.10.2 on Linux suffers a nasty problem with its use of mmap
and randomized address spaces
(https://bugzilla.redhat.com/show_bug.cgi?id=445545#c9) but it doesn't
seem like this is the same issue.

Rich.

--
Richard Jones
Red Hat

[Jean Krivine]

Dear all

I downloaded the last version of ocaml (3.10.2) but I must confess I
don't know what option I should pass to the compiler to make a binary
that uses 64 bits.
I tried naively ocamlopt -ccopt -arch -ccopt x86_64 but that doesn't
work. Any idea?

[Andres Varon]

Hello Jean,

There is no 64-bit native OCaml compiler for Mac OS X intel. I have a
patch that works in Leopard, but did not compile opt.opt in Tiger,
meaning that something is not OK, so I did not offer it to the
community. The bootstrap went fine, findlib and godi compiled OK too.
I can post the patches somewhere if you want to give it a shot.

My memory intensive application runs fine in Leopard with this
compiler. But the binaries do not execute in Tiger (I found that other
people had the same trouble copying a 64 bit apps from Leopard to
Tiger and the other way around, but didn't look into it).

If you want it ... I can post it, maybe someone can cleanup my job?
All that would be needed after patching is:

/configure -host x86_64-apple-darwin -prefix /opt/ocaml/experimental

(The prefix I always add for my ocaml-modified comilers).

best,

Andres

[Jean Krivine]

I'd be glad to try the patch if you could post it somewhere!

J

On Tue, Jul 15, 2008 at 3:31 PM, Andres Varon <ava...@gmail.com> wrote:
> Hello Jean,
>
> There is no 64-bit native OCaml compiler for Mac OS X intel. I have a patch
> that works in Leopard, but did not compile opt.opt in Tiger, meaning that
> something is not OK, so I did not offer it to the community. The bootstrap
> went fine, findlib and godi compiled OK too. I can post the patches
> somewhere if you want to give it a shot.
>
> My memory intensive application runs fine in Leopard with this compiler. But
> the binaries do not execute in Tiger (I found that other people had the same
> trouble copying a 64 bit apps from Leopard to Tiger and the other way
> around, but didn't look into it).
>
> If you want it ... I can post it, maybe someone can cleanup my job? All that
> would be needed after patching is:
>

> ./configure -host x86_64-apple-darwin -prefix /opt/ocaml/experimental

[Andres Varon]

On Jul 15, 2008, at 3:38 PM, Jean Krivine wrote:

> I'd be glad to try the patch if you could post it somewhere!

I have posted it in:

http://research.amnh.org/~avaron/ocaml/

best,

Andres

[Jean Krivine]

Great thanks!

J

[Jean Krivine]

Good news, I just tested the patch and it works great with my application!
I just had to modify the module random since a call to (Random.int
max_int) may raise and exception (it is made for 32 bits integers).
So I guess that modification should be included in the patch.

Thanks a lot Andres.
Jean

[Andres Varon]

On Jul 16, 2008, at 2:07 PM, Jean Krivine wrote:

> Good news, I just tested the patch and it works great with my
> application!
> I just had to modify the module random since a call to (Random.int
> max_int) may raise and exception (it is made for 32 bits integers).
> So I guess that modification should be included in the patch.

I don't think that's a good idea. You have to use Random.int64 to get
a 64 bit random integer. The Random.int function will return an
integer between 0 and 2^30. Check the Random module documentation here:

http://caml.inria.fr/pub/docs/manual-ocaml/libref/Random.html

I wouldn't play with a random number generator unless I know exactly
what I'm doing. Your results depend on it! (well, your messed-up-by-
andres compiler could already have issues ... :-(, for what I use it I
can verify the result with a 32 bit binary or a 64 bit linux binary,
if you can, then do the same!).

Andres

[Jean Krivine]

I agree. I should use Int64 instead of just int, but I still think
that the application (Random.int max_int) should not be exception
prone. Since max_int is architecture dependent, then so should be
Random.int no?

But you point is well taken. Thanks again
J