Make: Out of Memory error

[Daniel Jomphe]

I'm building Jazz on Leopard, with 4 GBs of memory, of which > 3 GBs
are free.

Prior to trying this out, I built gambit-c. Doing a `make check` on it
succeeded, although heartbeats were really off, as is expected from OS
X. I believe this isn't causing my issue, though.

What happens is the compilation process grows to 3 GBs of memory
usage, while my swap file grows from 0 to ~800 MBs. That's when the
build fails:

; compiling jazz/dialect/language/_language...
cc1(36691) malloc: *** mmap(size=116412416) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug

cc1: out of memory allocating 116410896 bytes after a total of 0
bytes
*** ERROR IN jazz.compile-source, "../lib/core/src/core/module/
builder/_builder.scm"@75.13
-- C compilation or link failed while compiling
"/Users/reniel/dev/tools/jazz/lib/jazz/src/jazz/dialect/
language/_language.jazz"
*** ERROR IN jazz.make-product, "../kernel/runtime/
kernel.scm"@1153.15 -- failed
failed

I experimented, like Paul Onions before me, with rebuilding gambit
using different configurations:

./configure --enable-single-host --enable-gcc-opts
./configure --enable-gcc-opts
./configure --enable-single-host

I know this isn't what solved Paul's issue, but who knows? None of
these configurations solved my issue.

Next thing I might want to try is using gambit 4.4.0's stable release
instead of the following one, what do you think?

commit 8592623c9bf6891def827b68716ca6e5e2c70c69
Author: Marc Feeley <fee...@iro.umontreal.ca>
Date: Wed Jul 1 09:51:13 2009 -0400

[Guillaume Cartier]

Many people have reported this problem.

For sure jazz.dialect.language has always been a very large file which
as a result implies a long and memory demanding compilation with GCC.
What I don't understand is what changed to make it jump to needing
more than 3G when you compile it easily on a machine with 2G? Sure
there was a couple new definitions added to language but this should
not result in such an explosion of the memory needed to compile it.

Also, Stéphane LeCornec did a full rebuilt today on Mac OS X that
completed successfully. Can you tell me what GCC version you are using
(gcc --version). Maybe it is a new optimisation in GCC that is
exponential in terms of code size.

In the next months we will be working on optimising the code generated
by the Jazz code walker. This should lower the memory requirements of
compilation by a big factor. In the meantime, I will split language
into a couple modules that can be compiled separately.

Guillaume

[Daniel Jomphe]

Ok, using the FSF's gcc 4.4, I was able to build (gambit and) jazz.
Apple's gcc 4.2 and 4.0 were the culprits here.

FSF's gcc often reports it doesn't recognize '-no-cpp-precomp', but
that's all.

Making jazz eats up to ~1.5G of memory, and takes 4m45s on my
late-2007 iMac (2.4 GHz).

Making jedi eats up to ~1.5G, and takes:
real 14m39.396s
user 16m0.217s
sys 1m39.301s

Both makes are asked to use 2 parallel jobs, although only the jedi
one seems to use the second core.

...You guys seem to have very fast MacBook Pros :)

On Jul 9, 8:37 pm, Guillaume Cartier <gcart...@jazzscheme.org> wrote:
> Many people have reported this problem.
>
> For sure jazz.dialect.language has always been a very large file which
> as a result implies a long and memory demanding compilation with GCC.
> What I don't understand is what changed to make it jump to needing
> more than 3G when you compile it easily on a machine with 2G? Sure
> there was a couple new definitions added to language but this should
> not result in such an explosion of the memory needed to compile it.
>
> Also, Stéphane LeCornec did a full rebuilt today on Mac OS X that
> completed successfully. Can you tell me what GCC version you are using
> (gcc --version). Maybe it is a new optimisation in GCC that is
> exponential in terms of code size.
>
> In the next months we will be working on optimising the code generated
> by the Jazz code walker. This should lower the memory requirements of
> compilation by a big factor. In the meantime, I will split language
> into a couple modules that can be compiled separately.
>
> Guillaume
>

[Marc Feeley]

On 10-Jul-09, at 9:52 AM, Daniel Jomphe wrote:

> Ok, using the FSF's gcc 4.4, I was able to build (gambit and) jazz.
> Apple's gcc 4.2 and 4.0 were the culprits here.
>
> FSF's gcc often reports it doesn't recognize '-no-cpp-precomp', but
> that's all.
>
> Making jazz eats up to ~1.5G of memory, and takes 4m45s on my
> late-2007 iMac (2.4 GHz).
>
> Making jedi eats up to ~1.5G, and takes:
> real 14m39.396s
> user 16m0.217s
> sys 1m39.301s
>
> Both makes are asked to use 2 parallel jobs, although only the jedi
> one seems to use the second core.
>
> ...You guys seem to have very fast MacBook Pros :)

If you want to speed the compilation then use a Gambit that was
configured with no options (except for --prefix=... and the like), i.e.

% ./configure

In this case the C files generated by the Gambit compiler will be
compiled in "multiple host" mode, which means that each Scheme
procedure will be compiled as a C function (i.e. the C compiler will
have multiple (small) C functions to compile). If you specify --
enable-single-host then a single (large) C function will contain the
whole group of Scheme procedures in the file. This usually generates
much faster code (factor of 2 is typical) but the C compiler will take
longer and require more memory. Unfortunately the gcc compiler has a
time/space complexity that is not very stable over successive version
of the compiler. So one version of gcc may compile the code in a
reasonable time, and the next may require so much memory that the C
compilation fails. Newer versions are not necessarily better.

Marc

[Guillaume Cartier]

The jazz.dialect.language module is now split into several modules.
The main one is still pretty big but should not cause any problems I'd
guess as its generated Scheme code size is similar to Gambit's biggest
Scheme lib files.

Guillaume

[Daniel Jomphe]

Very nice! Compilation space remains some hundred MBs higher than
Gambit's, but it's much better than before:

jazz/dialect/language/functional : 1.8G
jazz/ui/text/Text-View : 1.9G

Also, it looks like jazz now compiles more in parallel than before.

That said, the compilation now fails at this point:

; compiling jazz/ui/window/platform/x11...
*** ERROR IN jazz.walk-library, "../lib/core/src/core/library/syntax/
walker.scm"@2757.3 -- Walk problems encountered:
In jazz.ui.window.platform.x11
At translate-shortcut
Unresolved symbol: get-application
At get-toplevel
Unresolved symbol: get-application
At get-stage
Unresolved symbol: get-application
------------- REPL is now in #<thread #2> -------------
*** ERROR IN jazz.make-product, "../kernel/runtime/kernel.scm"@1153.15
-- failed

On Jul 13, 4:08 am, Guillaume Cartier <gcart...@jazzscheme.org> wrote:
> The jazz.dialect.language module is now split into several modules.
> The main one is still pretty big but should not cause any problems I'd
> guess as its generated Scheme code size is similar to Gambit's biggest
> Scheme lib files.
>
> Guillaume
>

[Daniel Jomphe]

I just built with the new fix for X11. Thanks.