[linux] thin archives and fd limit

[Evan Martin]

We recently committed a gyp change that makes us use "thin" ar
archives, where the .a file references the paths to all the .o files
rather than incorporating them into the .a file. This improves build
time a lot -- normally we build 700mb of object files, then copy them
into a 700mb libwebcore.a, which is then linked into the final binary;
each will flush your disk cache if you don't have a lot of RAM.

The change makes builds a *lot* faster.

Unfortunately, it seems likely this change caused a builder to fail:
LINK(target) out/Release/chrome
/usr/bin/ld: fatal error: out of file descriptors and couldn't close any
collect2: ld returned 1 exit status
make: *** [out/Release/chrome] Error 1

This might be a bug in gold:
http://sourceware.org/bugzilla/show_bug.cgi?id=10708
Or it might be that we're hitting a per-process fd limit (via ulimit).
Or we're hitting the system-wide fd limit due to running links in parallel.
- We can test this by examining dmesg (look for "VFS: file-max limit
<number> reached").
- We can work around this by doing links in serial.

I don't have access to the failing machine, and it of course works
locally (I'm using GNU ld with different hardware and a different
Linux release). Can anyone:
- examine the machine for the above dmesg output?
- see what ulimit says about file descriptor limits
- tell me how to tell what version of gold it's using?
- reproduce this problem with gyp trunk on their machine?

[Antoine Labour]

Works for me with gyp trunk +  gold ( 'GNU gold (GNU Binutils 2.20) 1.9' ).

Antoine

[Evan Martin]

An update.

After lots of experimentation, I verified that:
- gold < 2.20.1 produces bad binaries on Ubuntu Lucid
- gold 2.20.1 (the newest version) runs into this fd limit when
building release Chrome
- upping the file descriptor ulimit works around it, but you need to
be root to do that
- gold trunk works just fine
- normal ld works just fine in all configurations

Here is some link timing info with current trunk gyp and the gold trunk.
Debug Release
ld 40s 8s
gold 16s 2.5s (!)

Turning off the thin archive flag leaves the link times roughly the
same. The incremental build times are relatively much longer (because
we pay a lot more disk access (well, the same amount we had before
this patch) in generating the .a files), but I am sad to say I think
that is the best course until we have releases of all the tools
involved that work properly.

If you have a system with a good setup, I believe after my change you
can still get thin archives by setting
make ARFLAGS.target=crsT ARFLAGS.host=crsT [other args here]
and I will turn it back on after there is a new release of gold.

PS: I was surprised to discover that linking in release can take under
three seconds (computing dependencies etc pushes the incremental
rebuild time up another 7 seconds, though, and building the
intermediate .a files doesn't help either).

On Tue, Jun 29, 2010 at 5:24 PM, Evan Martin <ev...@chromium.org> wrote:

[Markus Gutschke]

On Wed, Jun 30, 2010 at 15:50, Evan Martin <ev...@chromium.org> wrote:

 - upping the file descriptor ulimit works around it, but you need to
be root to do that

Add this line to your "/etc/security/limits.conf":

  * hard nofile 4096

This will allow users to increase their ulimit settings by calling

  ulimit -n 4096

If that's still too complicated, I think the cost of increasing it for all programs is pretty low. So, we could raise both the hard and the soft limit at the same time. Or we could change our buildscripts to always run:

  ulimit -n `ulimit -H -n`

That would raise the limit just for the buildscript's children.

Markus