PSA: unit_tests target getting too big to link on x86

[Gabriel Charette]

tldr; unit_tests seldomly OOMs linking on x86

Hello everyone,

As we know the unit_tests target is a beast...

It turns out it's getting too big though, so big that it has problems linking on x86 sometimes.

For example it flaked this morning on Google Chrome Linux in 

http://build.chromium.org/p/chromium.chrome/builders/Google%20Chrome%20Linux/builds/22787/steps/compile/logs/stdio#error1

where it needed 1370286640 bytes (1.28 GB) to link and couldn't get them.

It doesn't flake too often yet, variable on amount of RAM available at link time, but this will get increasingly worst if action is not taken.

Should we:

1) Start splitting unit_tests into separate, more specific, targets?
2) Switch to x64...?!

3) Your suggestion...

Cheers,

Gab

[Robert Sesek]

We had this issue a few years ago when building on Mac OS X 10.5. The linker on 10.5 was 32-bit and was running out of RAM. The solution was to upgrade to 10.6, which had a 64-bit linker. Splitting unit_tests may be a good idea at some point, but I think it would require further componentization to be useful.

Historical threads about this on Mac:

https://groups.google.com/a/chromium.org/d/topic/chromium-dev/TWRA6s6vXqk/discussion

https://groups.google.com/a/chromium.org/d/topic/chromium-dev/_30QbOy7cjw/discussion

https://groups.google.com/a/chromium.org/d/topic/chromium-dev/cVFMQzQv8Ko/discussion

https://groups.google.com/a/chromium.org/d/topic/chromium-dev/iWMNLqcF9uA/discussion

rsesek / @chromium.org

 

3) Your suggestion...

Cheers,

Gab

--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev

[Reid Kleckner]

Yes, please make it smaller!  We hit this on the memory waterfall because we build debug+static: http://crbug.com/137180

It has snuck in because a lot of the debug builders use the shared library build for speed.

On Wed, Jul 25, 2012 at 2:07 PM, Gabriel Charette <g...@chromium.org> wrote:

--

[Rachel Blum]

Would it be worthwhile to entertain the notion of moving out tests with non-trivial execution duration? (I.e. if it takes 20 seconds, it probably shouldn't be a unit test...)

Rachel

[Darin Fisher]

How about we link unit_tests.exe against chrome.dll instead of statically linking the equivalent of chrome.dll into unit_tests.exe?  We could use a CHROME_EXPORT macro to facilitate this.

-Darin

[Ricardo Vargas]

On Wed, Jul 25, 2012 at 4:28 PM, Darin Fisher <da...@chromium.org> wrote:

How about we link unit_tests.exe against chrome.dll instead of statically linking the equivalent of chrome.dll into unit_tests.exe?  We could use a CHROME_EXPORT macro to facilitate this.

That basically requires adding the export macro to everything... and it has to be done in a single step (no way to partially link against something). It is simpler to "just" carve more modules out from chrome; that reduces the size of unit_tests.exe accordingly (but doesn't help the static build).

[Dirk Pranke]

On Wed, Jul 25, 2012 at 7:46 PM, Ricardo Vargas <rva...@chromium.org> wrote:
>
>
> On Wed, Jul 25, 2012 at 4:28 PM, Darin Fisher <da...@chromium.org> wrote:
>>
>> How about we link unit_tests.exe against chrome.dll instead of statically
>> linking the equivalent of chrome.dll into unit_tests.exe? We could use a
>> CHROME_EXPORT macro to facilitate this.
>
>
> That basically requires adding the export macro to everything... and it has
> to be done in a single step (no way to partially link against something). It
> is simpler to "just" carve more modules out from chrome; that reduces the
> size of unit_tests.exe accordingly (but doesn't help the static build).
>

My experience w/ content suggests the same thing.

-- Dirk

[Darin Fisher]

On Wed, Jul 25, 2012 at 7:46 PM, Ricardo Vargas <rva...@chromium.org> wrote:

On Wed, Jul 25, 2012 at 4:28 PM, Darin Fisher <da...@chromium.org> wrote:

How about we link unit_tests.exe against chrome.dll instead of statically linking the equivalent of chrome.dll into unit_tests.exe?  We could use a CHROME_EXPORT macro to facilitate this.

That basically requires adding the export macro to everything... and it has to be done in a single step (no way to partially link against something). It is simpler to "just" carve more modules out from chrome; that reduces the size of unit_tests.exe accordingly (but doesn't help the static build).

good points.  it seems like needing to support the static build implies needing to break up unit_tests.exe into a smaller number of executables.

-darin

[Erik Wright]

One of the objectives of the Browser Componentization project is to move parts of the browser into their own targets, creating corresponding standalone unit test targets along the way. This isn't happening over night, but should have a positive impact in the medium term.

[Michael Moss]

On the Linux side (32-bit, official build only), I submitted a
workaround to strip debug symbols from the unit_tests binary. This
greatly reduces mmap size and seems to have done the trick on the
chromium.chrome and official builders for now. We can revert it once
Componentization or other such fixes are in.

Michael