OOM while building ghc 9.4.5

[Ilias Tsitsimpis]

Hi team,

While preparing ghc 9.4.5 in Debian experimental, it fails to build with
OOM on mipsel [1].

The command that fails is:

$ "inplace/bin/ghc-stage1" -hisuf p_hi -osuf p_o -hcsuf p_hc -static -prof -H32m -O -lffi -optl-pthread -optc--param -optcggc-min-expand=5 -optc--param -optcggc-min-heapsize=4096 -Wall -this-unit-id ghc-9.4.5 -hide-all-packages -package-env - -i -icompiler/. -icompiler/stage2/build -Icompiler/stage2/build -icompiler/stage2/build/./autogen -Icompiler/stage2/build/./autogen -Icompiler/. -Icompiler/stage2/build/. -optP-DHAVE_INTERNAL_INTERPRETER -optP-DCAN_LOAD_DLL -optP-include -optPcompiler/stage2/build/./autogen/cabal_macros.h -package-id array-0.5.4.0 -package-id base-4.17.1.0 -package-id binary-0.8.9.1 -package-id bytestring-0.11.4.0 -package-id containers-0.6.7 -package-id deepseq-1.4.8.0 -package-id directory-1.3.7.1 -package-id exceptions-0.10.5 -package-id filepath-1.4.2.2 -package-id ghc-boot-9.4.5 -package-id ghc-heap-9.4.5 -package-id ghci-9.4.5 -package-id hpc-0.6.1.0 -package-id process-1.6.16.0 -package-id stm-2.5.1.0 -package-id template-haskell-2.19.0.0 -package-id terminfo-0.4.1.5 -package-id time-1.12.2 -package-id transformers-0.5.6.2 -package-id unix-2.7.3 -Wall -Wno-name-shadowing -Wnoncanonical-monad-instances -Wnoncanonical-monoid-instances -this-unit-id ghc -XHaskell2010 -XNoImplicitPrelude -XBangPatterns -XScopedTypeVariables -XMonoLocalBinds -XTypeOperators -Rghc-timing -Wcpp-undef -Wincomplete-uni-patterns -Wincomplete-record-updates -no-user-package-db -rtsopts -O0 -Wnoncanonical-monad-instances -outputdir compiler/stage2/build -c compiler/./GHC/Hs/Instances.hs -o compiler/stage2/build/GHC/Hs/Instances.p_o -dyno compiler/stage2/build/GHC/Hs/Instances.dyn_o

virtual memory exhausted: Cannot allocate memory
`gcc' failed in phase `C Compiler'. (Exit code: 1)

Building an unregisterised ghc compiler on 32-bit targets has trouble
handling the Instances.hs file. This has been reported upstream here[2].
Until now, we have managed to workaround this by compiling this file
alone with O0 (as shown in the above command), but since version 9.4.5
even that doesn't work.

Is there anything else we can try to compile ghc 9.4.5 in mipsel, or
should we give up in supporting mipsel at all?

[1] https://buildd.debian.org/status/package.php?p=ghc&suite=experimental
[2] https://gitlab.haskell.org/ghc/ghc/-/issues/17048

PS please CC me as I am not subscribed to this list.

Best,

--
Ilias

[Jeffrey Walton]

Hi Ilias,

Speaking as someone who helps maintain C/C++ libraries; not someone
who packages for Debian or administers the mipsel gear...

I used to run into this alot with the Crypto++ library on ARM gadgets.
"Gadgets" are what I call the dev boards and IoT boards. The dev
boards were low resource, and often had 512MB or 1GB of RAM with no
swap file because of a SDcard. One particular file used to cause an
OOM kill more frequently than I care to remember.

I found two things helped or fixed the problem. First, I would build
with only one make job. That is, 'make -j 1'. 'make -j 3' or 'make -j
5' was sure to break the build.

Second, I would add a swap partition that provided enough space to run
the compiler. For a dev board with 512MB of RAM, I would setup a 2GB
swap partition. I would also set swappiness to 1 or 2 to keep things
in memory. zram would not help because even compressed, the compiler
needed a fair amount of space.

And I do burn through a lot of SDcards. But a new SDcard once or twice
a year is a better outcome than a DoS as the OOM Killer keeps killing
my jobs.

Jeff

[Ilias Tsitsimpis]

Hi Jeffrey,

On Wed, Jul 26, 2023 at 01:02PM, Jeffrey Walton wrote:
> [...]

> I found two things helped or fixed the problem. First, I would build
> with only one make job. That is, 'make -j 1'. 'make -j 3' or 'make -j
> 5' was sure to break the build.
>
> Second, I would add a swap partition that provided enough space to run
> the compiler. For a dev board with 512MB of RAM, I would setup a 2GB
> swap partition. I would also set swappiness to 1 or 2 to keep things
> in memory. zram would not help because even compressed, the compiler
> needed a fair amount of space.

Thanks for the hints. Unfortunately none of them have worked so far. I
am trying to compile just this file and nothing else, and I am using a
host with 8G of RAM and 8G of Swap. The problem seems to be the fact
that mipsel is limited to only 2G RAM space for user processes.

Unfortunately I cannot see a way forward here, other than removing ghc
(and every Haskell program) from mipsel.

--
Ilias

[Jeffrey Walton]

Hi Ilias,

On Fri, Jul 28, 2023 at 4:27 PM Ilias Tsitsimpis <ilia...@debian.org> wrote:
>
> On Wed, Jul 26, 2023 at 01:02PM, Jeffrey Walton wrote:
> > [...]
> > I found two things helped or fixed the problem. First, I would build
> > with only one make job. That is, 'make -j 1'. 'make -j 3' or 'make -j
> > 5' was sure to break the build.
> >
> > Second, I would add a swap partition that provided enough space to run
> > the compiler. For a dev board with 512MB of RAM, I would setup a 2GB
> > swap partition. I would also set swappiness to 1 or 2 to keep things
> > in memory. zram would not help because even compressed, the compiler
> > needed a fair amount of space.
>
> Thanks for the hints. Unfortunately none of them have worked so far. I
> am trying to compile just this file and nothing else, and I am using a
> host with 8G of RAM and 8G of Swap. The problem seems to be the fact
> that mipsel is limited to only 2G RAM space for user processes.

Oh, that's an interesting twist. But I see it could make sense since
this is a 32-bit platform.

> Unfortunately I cannot see a way forward here, other than removing ghc
> (and every Haskell program) from mipsel.

I'm spit-balling here, since I don't recall doing this in C/C++ for
this problem... Based on:

virtual memory exhausted: Cannot allocate memory
`gcc' failed in phase `C Compiler'. (Exit code: 1)

Break that source file up into two or three more manageable pieces.

If I am parsing the command line correctly, the source file is
Instances.hs and the output file is Instances.p_o. I would try to
create Instances_1.hs, Instances_2.hs and Instances_3.hs, and then let
the linker combine their object files later.

What I don't know is, is it possible to do that in Haskell. Some
languages are picky about things like that. C# and Java come to mind.

Jeff

[Ilias Tsitsimpis]

Hi Jeff,

On Fri, Jul 28, 2023 at 05:09PM, Jeffrey Walton wrote:
> Break that source file up into two or three more manageable pieces.
>
> If I am parsing the command line correctly, the source file is
> Instances.hs and the output file is Instances.p_o. I would try to
> create Instances_1.hs, Instances_2.hs and Instances_3.hs, and then let
> the linker combine their object files later.
>
> What I don't know is, is it possible to do that in Haskell. Some
> languages are picky about things like that. C# and Java come to mind.

This was suggested as a solution by upstream as well (see [1]) but they
found it was too difficult to do and ended up using to -O0 instead.

[1] https://gitlab.haskell.org/ghc/ghc/-/issues/18256

--
Ilias

[Jeffrey Walton]

The 18256 issue looks like it is a different complaint. 18256 tries to
speed up compile time. An OOM kill is a different problem.

Looking at the original command, the project is using -O, which is
-O2. Maybe you can drop it to -O1 -fno-inline.

>From Crypto++, I've found there's no material difference between -O1,
-O2 or -O3. Crypto++ provides SSE/AVX/NEON/ASIMD/Altivec acceleration
where it matters, so compiler optimizations make almost no difference
at runtime according to benchmarks.

I see Debian has https://wiki.debian.org/ReduceBuildMemoryOverhead,
but it does not look too helpful. (I updated with your discussion).

[Jeffrey Walton]

Hi Ilias ,

On Sat, Jul 29, 2023 at 2:44 AM Ilias Tsitsimpis <ilia...@debian.org> wrote:
>

Since the compile is failing in preprocessing, I took a look at GCC
preprocessor options.[1]

These look like they could help:

* -ftrack-macro-expansion=0
- Don't track macro expansion. Saves memory
* -no-integrated-cpp
- invoke cc1 twice, once for preprocessing and
once for actual compilation of the preprocessed input
* -fmax-include-depth=20
- Normally 200, don't allow deep nesting

And if it is an option, then:

* -fno-large-source-files

-flarge-source-files is an option, but I am not sure about the -fno-*
variant in this case.

Finally, this may help, but I don't understand it. I have never used it:

* -Mno-modules
- Disable dependency generation for compiled module interfaces.
Jeff

[1] https://gcc.gnu.org/onlinedocs/gcc/Preprocessor-Options.html

[Ilias Tsitsimpis]

On Sun, Jul 30, 2023 at 12:02AM, Jeffrey Walton wrote:
> Since the compile is failing in preprocessing, I took a look at GCC
> preprocessor options.[1]

Thanks for taking a look. Do you have any indication it fails in
preprocessing. Given how -O0 used to work, I would guess it fails during
compilation phase.

> These look like they could help:
>
> * -ftrack-macro-expansion=0
> - Don't track macro expansion. Saves memory
> * -no-integrated-cpp
> - invoke cc1 twice, once for preprocessing and
> once for actual compilation of the preprocessed input
> * -fmax-include-depth=20
> - Normally 200, don't allow deep nesting
>
> And if it is an option, then:
>
> * -fno-large-source-files
>
> -flarge-source-files is an option, but I am not sure about the -fno-*
> variant in this case.
>
> Finally, this may help, but I don't understand it. I have never used it:
>
> * -Mno-modules
> - Disable dependency generation for compiled module interfaces.

I can try those when I find some time, thanks.

--
Ilias

[Ilias Tsitsimpis]

On Sat, Jul 29, 2023 at 01:35PM, Jeffrey Walton wrote:
> The 18256 issue looks like it is a different complaint. 18256 tries to
> speed up compile time. An OOM kill is a different problem.

I agree. I linked to it because I wanted to point out that splitting the
GHC.Hs.Instances module is not an option, as noted in the issue.

>
> Looking at the original command, the project is using -O, which is
> -O2. Maybe you can drop it to -O1 -fno-inline.

If you notice towards the end of the command, we pass -O0, which
overrides the first -O. This used to work (and it's still useful for the
other 32-bit architectures) but doesn't work any more on mipsel.

--
Ilias