Object file name mapping and complete_static_lib

[Sylvain Defresne]

As part of building Chrome on iOS we want to build a static library that is self-contained so that we can share it with other teams.

With gyp this required a small hack to collect the list of all object files but then worked fine. With gn, the plan was to just create a static library target with "complete_static_lib = true". However, we get failures like this:

libtool: warning same member name (url_util.o) in output file used for input files:
obj/net/net/url_util.o and: obj/url/url/url_util.o (due to use of basename, truncation, blank padding or duplicate input files)

This is due to the fact that gyp included the target name into the generated object file, while gn does not (as explained in this email).

We need to ship that library as a static library (dynamic library is not really an option on iOS in that case), and I don't think there is no way to force libtool to uniquify the names when creating the static library. Thus, I'd like to change the name of the generated object files when using gn at least for the iOS and Mac toolchain.

Would there be any objection to using {{target_name}}.{{source_name_part}}.o for the object files when building for iOS or Mac? If not, should I also make the change on other platforms (it could be problematic on Windows as MAX_PATH is limited to 260 characters IIRC).

WDYT?

-- Sylvain

[Andrew Grieve]

I recently ran into the case with adding brotli to third-party where I wanted to have two targets in the same BUILD.gn file compile the same source file.

https://code.google.com/p/chromium/codesearch#chromium/src/third_party/brotli/BUILD.gn

For a while we were having trouble with the "bro" executable not being happy with sanitizers enabled. Since this is just a tool for building, I wanted to disable sanitization for it by having it include the .cc files listed in the source_set directly. This same-naming problem prevented that. (in the end we skirted the issue by editing the bro source code and removing the try/catch, which was the cause of sanitizers being unhappy).

So, a bit obscure, but this is maybe another motivation for adding the target_name to the .o files.

[Dirk Pranke]

Here's the discussion around this from four years ago w/ GYP that led to GYP's conventions ...

https://groups.google.com/forum/#!topic/gyp-developer/VAfuE-H553Y

Personally, this seems like a bug in libtool that should be fixed, and I'd work around it in this case by renaming one of the two

files, but I can see the argument for changing the naming scheme as well.

-- Dirk

[Roland McGrath]

On Wed, Jun 1, 2016 at 2:58 PM, Dirk Pranke <dpr...@chromium.org> wrote:
> Here's the discussion around this from four years ago w/ GYP that led to
> GYP's conventions ...
>
> https://groups.google.com/forum/#!topic/gyp-developer/VAfuE-H553Y
>
> Personally, this seems like a bug in libtool that should be fixed, and I'd
> work around it in this case by renaming one of the two
> files, but I can see the argument for changing the naming scheme as well.

I can't speak directly for OSX's libtool, but the traditional behavior
of Unix ar is to drop leading directory names from the names stored in
the archive, so foo/x.o and bar/x.o both go into the archive as just
"x.o". ar does let you have multiple entries in an archive that have
the same name, and that file name doesn't really matter to linking.
So while it generally does work to have both a foo/x.o and a bar/x.o
go into the same archive, it can be confusing in ways that might be
deemed to violate the principle of least astonishment (e.g. "mkdir x;
cd x; ar x ../foo.a; ar cr ../new-foo.a *" removes a file from the
archive, where on first blush one would expect it to be a no-op).

[Dirk Pranke]

Ayup. And traditionally the way you avoid problems (at least in my experience)

is to either keep your libraries relatively small, or require unique basenames

for source files in your project, or both :). 

-- Dirk

[Brett Wilson]

In GN, unlike GYP, the target name is very non-unique. We have over 100 targets named "test_support" and a whole bunch named things like "browser".

So this probably doesn't solve the problem the way you want. When I initially moved to the current naming scheme, there were a few naming conflicts that I fixed. It was annoying but better than having to encode either the full target path or a hash of it in every name.

Brett 

[Sylvain Defresne]

I understand that even if we include the target name in the object file name (i.e. use obj/net/net/net.url_util.o) there may still be collision as there are many targets with the same name (as you mention the "test_support" targets fits in that category).

Renaming the files is a solution that is difficult to sustain however as the target is build downstream (thus cannot turn a bot red when a CL introducing a collision lands). Even with the previously mentioned caveat, including the target name in the object file name would make the name unique enough for us because our library does not depends on any test support library (it is a production library with "testonly = false") and it only depends on //ios/web and below.

If including the name in the library is not an option, and since fixing libtool is not possible (at least in the short term since we are using the version of libtool coming from Xcode for linking on iOS), can I add an option to static_library target to change the filenames before invocation of libtool? Something like this:

static_library("foo_standalone") {

  deps = [ ... ]

  complete_static_lib = true

  ensure_unique_object_filenames = true

}

and then this pass a flag to the "alink" tool. The "alink" tool can then implement this however it wants. On iOS and Mac, I can then invoke the libtool wrapper and have rename the file to have unique names by creating symlinks, i.e. instead of this:

libtool -static -o libfoo_standalone.a obj/ios/web/web/url_util.o obj/url/url/url_util.o

then we'd end up with the wrapper doing

ln -s obj/ios/web/web/url_util.o obj:ios:web:web:url_util.o

ln -s obj/url/url/url_util.o obj:url:url:url_util.o

libtool -static -o libfoo_standalone.a obj:ios:web:web:url_util.o obj:url:url:url_util.o

If we consider this a bug of libtool, maybe we can even drop the "ensure_unique_object_filenames" flag and always perform the mangling on iOS/Mac in the libtool wrapper if "complete_static_lib" is true.

WDYT?

-- Sylvain

[Václav Brožek]

A side information which was not apparent from my original report: url_util.o was just an example, there were about 30 other conflicts between different directories (//base, //net, etc.). That would also speak against renaming individual files to fix this (although Sylvain's argument about downstream seems good enough already).

Cheers,

Vaclav

[Brett Wilson]

I would prefer to add a {{target_hash}} pattern that a tool chain could use than your proposal: it seems more conceptually simple.

How much stuff does this static library cover?

Brett 

[Sylvain Defresne]

Mostly //ios/web and it's dependencies (//base, //net, //url, //crypto, some third_party libraries).

I was reminded today that we have need for such a library upstream too (and that I forgot to convert it to gn). So I'm going to create an upstream CL to convert that library (crnet_standalone) to gn so that we can discuss on a proper CL.

-- Sylvain

[silvi...@gmail.com]

Is ensure_unique_object_filenames support in GN ?

Silviu

[Dirk Pranke]

There is nothing like that in GN itself, since GN doesn't impose any sort of naming scheme.

The //build files that Chromium uses do not ensure unique filenames, but one could certainly write other toolchain rules that did.

-- Dirk

[Henrik Kjellander]

I assume this is still unchanged on Mac? The only workaround for this problem is to ensure all filenames are unique? 

I'd like to create a complete_static_lib for WebRTC (https://bugs.webrtc.org/6418) but this limitation is stopping me from doing so (but I can at least do the other platforms).

[Brett Wilson]

Correct, nothing has changed in this area.

Brett