DELETE MOAR CODE
> --
> Chromium Developers mailing list: chromi...@chromium.org
> View archives, change email options, or unsubscribe:
> http://groups.google.com/a/chromium.org/group/chromium-dev
>
-scott
On Mon, Jun 28, 2010 at 11:30 AM, Robert Sesek <rse...@chromium.org> wrote:
I will remove some of my apps experiments that didn't work out. I
don't think it is enough to help, but it's the thought that counts
(right?).
- a
--
Unfortunately, the issue is mapping the various libraries involved at
link time. Breaking up unit_tests might only buy you a week or two
before you hit another problem mapping things.
Hmm. Another possibility on the memory fragmentation angle would be
to put the unit_test object files into a library, and then link
against it. That way instead of 400 or so .o files scattered about,
there would be another half-gig .a file.
-scott
On Tue, Jun 29, 2010 at 6:29 AM, Timur IskhodzhanovUnfortunately, the issue is mapping the various libraries involved at
<timu...@chromium.org> wrote:
> Here are a few bugs open:
> http://code.google.com/p/chromium/issues/detail?id=45207 (Mac issues)
> http://code.google.com/p/chromium/issues/detail?id=44241 (similar issues on
> Linux 32-bit build)
> Can we split unit_tests into two or more parts?
link time. Breaking up unit_tests might only buy you a week or two
before you hit another problem mapping things.
Hmm. Another possibility on the memory fragmentation angle would be
to put the unit_test object files into a library, and then link
against it. That way instead of 400 or so .o files scattered about,
there would be another half-gig .a file.
-scott
That’s problematic, especially for tests, because there won’t be any
reason for the linker to actually look at the .o objects inside the
test code .a archive—they won’t resolve any undefined symbols. The Mac
linker doesn’t have -whole-archive/-no-whole-archive, it only has
-all_load, which can only be enabled for an entire link session and
can’t be individually enabled or defeated for specific .a archives.
I appreciate all of the suggestions people are offering, but we’ll be
moving to 10.6 as a build platform with its 64-bit linker. I’ll send
out an e-mail about this later today for those that weren’t present
when we discussed this yesterday. We’ve invested enough time and
energy in band-aids already, and each “success” only buys a few days,
so I think it’s time to bite the bullet and move on.
Timur, we can continue testing with Valgrind on 10.5 by ferrying
builds done on 10.6 between machines.
Mark
In seriousness, on IRC we poked at the size buildbots and noticed that
turning on NSS on Mac gained us another 1.4 pounds -- er, megabytes.
I wonder if you could quickly turn that off until Mark's (better) 10.6
plan comes to fruition.
Not gonna help. :(
That’s an interesting idea, but I think we’re at—beyond, actually—the
point of diminishing returns. libwebcore.a is a beast, at over 1GB in
a debug build from a few days ago. One of the existing band-aids has
us loading it while the address space is relatively wide open.
libbrowser.a, second-heaviest at over 600MB, was formerly the first
library on the link line, and is now #2. Fragmentation is a big
problem, but with these numbers, we’re kind of pushing the limits in a
more absolute sense, too.
This may be worthwhile if other platforms up against this same problem
can’t move to 64-bit linkers, but it’s probably not going to do too
much for us for too long.
Mark
For things where you know "Link all of these", instead of a library
you could use ld's:
-r Merges object files to produce another mach-o object file
with file type MH_OBJECT.
Seems to work alright when I snag the link command-line and bust
things up manually. We have a really large number of items on our
link command-line :-).
> I appreciate all of the suggestions people are offering, but we’ll be
> moving to 10.6 as a build platform with its 64-bit linker. I’ll send
> out an e-mail about this later today for those that weren’t present
> when we discussed this yesterday. We’ve invested enough time and
> energy in band-aids already, and each “success” only buys a few days,
> so I think it’s time to bite the bullet and move on.
This is surely the correct overall solution, but in the meanwhile I'm
nervous about whether we'll lack coverage for a period. The 10.6 bots
on the waterfall provide some, but not having trybots for any length
of time could get annoying!
-scott
One last workaround to mention: shared library link. Build a
webcore.dylib or whatever. Would require some gyp surgery and dealing
with build breaks occasionally.
Victor is already doing the windows part so it'll eventually be
(relatively) easy to do on the Mac.
Cannot we use 64bit linker on 10.5? I've tried this idea on my 10.5
and it seems to be OK.
If you trust me, you can download the following file, put it into
/Developer/usr/bin, and see if it works.
I created the above file by
1. download the source code of ld64 from apple's website
% wget http://www.opensource.apple.com/tarballs/ld64/ld64-85.2.1.tar.gz
% tar -xvzf ld64-85.2.1.tar.gz
2. fix project.pbxproj
% cat ld64.patch
diff -ur ld64-85.2.1.orig/ld64.xcodeproj/project.pbxproj
ld64-85.2.1/ld64.xcodeproj/project.pbxproj
--- ld64-85.2.1.orig/ld64.xcodeproj/project.pbxproj 2010-06-29 07:54:08.0000
00000 -0700
+++ ld64-85.2.1/ld64.xcodeproj/project.pbxproj 2010-06-29 10:41:27.000000000 -0
700
@@ -552,7 +552,7 @@
PREBINDING = NO;
PRODUCT_NAME = ld;
SECTORDER_FLAGS = "";
- VALID_ARCHS = "i386 ppc";
+ VALID_ARCHS = "i386 ppc x86_64";
VERSIONING_SYSTEM = "apple-generic";
WARNING_CFLAGS = "-Wall";
};
@@ -712,7 +712,7 @@
INSTALL_PATH = /usr/bin;
PREBINDING = NO;
PRODUCT_NAME = rebase;
- VALID_ARCHS = "i386 ppc";
+ VALID_ARCHS = "i386 ppc x86_64";
};
name = Release;
};
% patch -p0 < ld64.patch
3. build the linker
% xcodebuild ARCHS=x86_64 -project ld64.xcodeproj -configuration Release
(note that the build of "rebase" seems to fail... but we don't need this binary)
4. put it into /Developer/usr/bin
% sudo cp build/Release/ld /Developer/usr/bin
Obviously, ld64-85.2.1 doesn't support x86_64 officially so the output
of the 64bit ld can be broken.
Thanks,
That’s what makes this pretty scary.
In my work with getting the 10.5/Xcode 3.1 toolchain running on Linux,
I found that significant parts would not build or work correctly for
x86_64. ld64-85.2.1 was not among those parts, but it depended on
components in other packages that had problems. I consider this
approach to be “shaky ground.”
If it works for you and others, that’s awesome, but I don’t think we
can really take this approach for production builds (including bots).
Mark
Have you tried this with unit_tests in a debug build that wouldn’t
link? Did it make it linkable?
> We have a really large number of items on our
> link command-line :-).
Yeah, and the command line doesn’t even have the .o files in it,
they’re listed separately in a file. Huge, huge command line anyway.
Mark
Yes. What I did:
- reenable svg for gyp.
- gclient sync.
- build target unit_tests.
- failed with mmap error.
- LINKFILE=xcodebuild/chrome.build/Debug/unit_tests.build/Objects-normal/i386/unit_tests.LinkFileList
- egrep unittest $(LINKFILE) | xargs ld -ObjC -o uber.o -r
- egrep -v unittest $(LINKFILE) > unit_tests.LinkFileList
- echo uber.o >>unit_tests.LinkFileList
- re-run original gcc line with $(LINKFILE) replaced by unit_tests.LinkFileList
Even passed a test! Unfortunately, when I re-enabled svg to capture
the command-line, I noticed that a bunch of other stuff didn't look
like it was linking right anymore. AFAICT some don't have a huge
LinkFileList (Chromium Framework.LinkFileList lists three files). So
this might only work for a little while, sigh.
Unless we add another layer to aggregate the .a files into an uber.a
file ... and then rewrite ld entirely at some point. Oh yeah.
-scott
Right, this wouldn’t really work for the framework (chrome_dll), so
it’ll only buy some time for unit_tests.
If you wanted to clean this up as a final band-aid for unit_tests,
that’s OK. Of all of the band-aids we’ve seen so far, this one is the
least hackalicious, although the GYP/Xcode work will get a little
“cute.”
GYP r833 (yesterday) added support for listing .o files in 'sources'
sections for the Xcode generator. If you wanted to do this, you’d
probably need to make unit_tests a none-type target on the Mac (and
also conditionally rename it), then hang a postbuild off of it to
build a monster .o file (would "ld -r -all_load
libunit_tests_monster.a -o unit_tests_monster.o" work? if not, use ar
x to get a bunch of .o files and drop the __.SYMDEF) and then have
(again, conditionally on the Mac) a unit_tests target that depends on
unit_tests_monster and lists unit_tests_monster.o in its sources list.
The new target would also need a dummy .mm file (which could be empty)
to actually convince Xcode to link, and to do it with the right
support libraries for Obj-C++.
Mark
Any progress on this?Mac Valgrind can't build chrome and ui_tests right now...I know we're moving towards 10.6 but Valgrind is still 10.5-only(Alexander is working on Valgrind for 10.6)Can anyone take a look and maybe apply yet another bandaid to the Mac UI Valgrind bots?..
Timur Iskhodzhanov,
Google Russia
Has anyone tried breaking up libwebcore.a?