serialize/limit link step on OSX

[David Levin]

On my Mac, I noticed that my builds of "all" tend to end with about 9 items being linked in parallel. It takes a long time and my machine becomes unusable.

Taking a hint from Albert's email earlier this week, I wrote a little script to fix this. At the moment, I only allow 4 links in parallel (not really fine tuned but it works well enough to scratch my itch). It keeps my machine responsive and results in a faster build overall.

Here's what I did.

# Save the real linker.

sudo cp /Developer/usr/bin/ld /Developer/usr/bin/real-ld

# Create a script file to limit the number of linkers run in parallel.

echo -e '#!/bin/sh\nlockfile=${TMPDIR}/link.lock\n\ncount=1;\nwhile !(shlock -f "${lockfile}${count}" -p $$) do\n  let count=count+1;\n  if [ ${count} -eq 5 ]; then\n    let count=1;\n    sleep 1;\n  fi\ndone\n\nexec /Developer/usr/bin/real-ld "$@"\n\nrm "${lockfile}${count}"' >  ${TMPDIR}/junk.txt

# Replace the linker with the script.
sudo cp ${TMPDIR}/junk.txt /Developer/usr/bin/ld

rm  ${TMPDIR}/junk.txt

sudo chmod a+x /Developer/usr/bin/ld

If there is a simpler way to do it, please speak up.

Best wishes,

dave

PS I haven't yet added this to http://code.google.com/p/chromium/wiki/MacBuildInstructions in case there is something I'm missing.

[Paweł Hajdan, Jr.]

Shouldn't we fix it right in the codebase / build scripts?

--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev

[David Levin]

On Fri, Jul 16, 2010 at 10:25 AM, Paweł Hajdan, Jr. <phajd...@chromium.org> wrote:

Shouldn't we fix it right in the codebase / build scripts?

Well I just finished writing it early this morning :) and I'm sharing quickly in case it is helpful. It doesn't have any vetting (maybe there is a better way). I'm not sure that it improves things for everyone or that it is the right approach.

But if folks find it useful, then that seems like the next logical step. (I also don't know how possible it is to integrate this into the xcode build.)

dave

[Drew Wilson]

I've never noticed my machine even slowing down during a link. I have a pretty beefy Mac Pro (8-core, 12 GB DDR3, separate hard drive for source), so I'd prefer not to change the build process in a way that's going to lead to slower builds.

-atw

[Mark Mentovai]

David Levin wrote:
> On my Mac, I noticed that my builds of "all" tend to end with about 9 items
> being linked in parallel. It takes a long time and my machine becomes
> unusable.

Wow! I’ve never really seen more than two targets being linked
simultaneously. Or maybe that’s four? It’s certainly less than ncpus
on my beefy Mac Pro. I guess disk seeking still might be a problem.
I’m a little surprised you’re actually seeing 9 in parallel, though.
Are they actually all running ld, or is Xcode holding any of them
until it has enough local “slots” free?

Mark

[David Levin]

They all appear to be running ild,and my machine becomes unusable, and it takes many, many minutes to link. No idea why.

No worries. I have a good enough fix for me and apparently no one else needs it :). I must have done some weird setting somewhere.

dave

[Dmitry Titov]

After my 12Gb Mac started to use up all the memory for 8-9 ld processes (and some compiler processes on top of that) and go into hopeless swapping, I've used Dave's script with much success. YMMV.

Which makes me think 24Gb of RAM for Snow Leopard might be a great idea...

[Jeremy Orlow]

For the first time in a while, I tried to build the chrome target "all" on my mac.  After a few minutes, my 12gb ram mac pro became unusable (until I force quitted xcode).

I also tried doing a full build on my laptop this weekend.  I started it before going to bed.  In the morning, the machine was completely unresponsive even after force quitting everything and letting it sit for a bit.

Both of these were snow leopard machines running the latest xcode.  

+1 to Pawel's suggestion to integrate something like this into the build itself.  (I would myself, but I really know nothing about how GYP or xcode itself works.)

J

[Nico Weber]

Just ran into this yesterday, too. Thanks for the script, Dave.

Nico

[Mihai Maerean]

Apart from the ld that exists in /Developer/usr/bin, there's a 2nd ld in /usr/bin/ so these steps should be performed on that one also:

# Save the real linker.

sudo cp /usr/bin/ld /usr/bin/real-ld

# Create a script file to limit the number of linkers run in parallel.

echo -e '#!/bin/sh\nlockfile=${TMPDIR}/link.lock\n\ncount=1;\nwhile !(shlock -f "${lockfile}${count}" -p $$) do\n  let count=count+1;\n  if [ ${count} -eq 5 ]; then\n    let count=1;\n    sleep 1;\n  fi\ndone\n\nexec /usr/bin/real-ld "$@"\n\nrm "${lockfile}${count}"' >  ${TMPDIR}/junk.txt

# Replace the linker with the script.

sudo cp ${TMPDIR}/junk.txt /usr/bin/ld

rm  ${TMPDIR}/junk.txt

sudo chmod a+x /usr/bin/ld

Mihai

[Nico Weber]

This thread is almost 2 years old. If you use ninja, ld will be
serialized automatically, so this script should no longer be
necessary.

Nico