Bundling dependent libraries on macOS

[Andrew Kaster]

Hi GN folks,

I'm working on an experimental GN build for the Ladybird web browser, partially motivated by a more straightforward way to bundle dependencies and resources than CMake seemed to offer.

Nico has reviewed some of the changes here already:

https://github.com/SerenityOS/serenity/pull/18663

I'm working to clean that up, but I have another set of changes on a local branch here to actually create an app bundle for macOS: 

https://github.com/ADKaster/serenity/commit/8d560dde63b2289507afe2caaf5ca5b02fe7be56#diff-759cbd162bb7e146f83ad5260f8c9e82b6cf75952a9f7cde6cf5a92f283daab7R151

The thing that's most concerning to me is that I have to express the list of shared library dependencies in three places now:

- In the dependencies of the libraries that the executables depend on (the normal gn build)

- In the deps of my "bundle_libs" rule

- With the actual build location of the libs and their final output name in the sources of the "bundle_libs" rule

Nico mentioned to me that the chromium build has a strategy of creating independent shared libraries for developer builds, but one big shared library for release, which makes the bundle step a bit more straightforward.

Is there a straightforward way to not have to repeat the libraries so much? So far I can think of:

- Just keep it as-is, and repeat all the libs in the bundle_data (twice!)

- Create a shared_library template wrapper that does the same thing chromium does and create one big shlib for release that gets put in the bundle.

-  Create a python script that walks the output of otool -L  and generates the list as part of the build, just before bundling the data. This is similar to what the CMake build currently does by recursively walking the LINK_LIBRARIES property of each executable/library and deduplicating the lists at the end.

Thanks,

Andrew Kaster

[Dirk Pranke]

Do you care about a Windows port? 

Assuming Nico is referring to the thing I think he's referring to (component builds), there is some amount of overhead in actual C++ files to do it: you have to be able to change how the export declarations work for symbols that need to be exported out of a shared library. I would probably not approach this lightly.

But, if you don't care about Windows (or maybe think it might be something you can worry about later), component build isn't too complicated.

Unfortunately, I can't say I'm all that familiar with how bundles work in GN and so I don't have any real advice there.

-- Dirk 

--
To unsubscribe from this group and stop receiving emails from it, send an email to gn-dev+un...@chromium.org.

[Andrew Kaster]

Windows is definitely a "later" concern at the moment. I tried to get some groundwork for a windows port into the project a few months ago and the response from other maintainers was "ehh, show us the whole browser working and that the maintenance won't be crazy high first". There's some prototypes floating around but they definitely need polish before being considered for actual support.

Andrew

[Ben Boeckel]

On Fri, May 12, 2023 at 11:12:03 -0700, Andrew Kaster wrote:
> - Create a python script that walks the output of otool -L and generates
> the list as part of the build, just before bundling the data. This is
> similar to what the CMake build currently does by recursively walking the
> LINK_LIBRARIES property of each executable/library and deduplicating the
> lists at the end.

Note that CMake has `file(GET_RUNTIME_DEPENDENCIES)` which now walks the
metadata in the libraries themselves. This helps to ensure that library
references in the binaries are actually correct. It's basically a port
of the first part of these scripts (`fixup_bundle.*.py`):

https://gitlab.kitware.com/paraview/common-superbuild/-/tree/master/cmake/scripts

where the second part is not yet implemented (fixing binary references
once bundled to point to each other consistently).

Note that I do *not* recommend using `otool -L` (or `ldd`) for the
following reasons:

- cross-compile is harder as you now need to juggle `--sysroot`-style
bits in cooperation with the dynamic loader tool in use (and massage
paths afterwards)
- you do not know by what mechanism each path was found (e.g., the above
scripts explicitly ignore things like `DYLD_LIBRARY_PATH` because the
libraries should work on their own) and therefore how to "fixup" each
intra-library reference (and library id on macOS) is not well-defined)
- it is recursive (you want to be able to stop at certain libraries that
you expect to just be there in general and ignore *their*
dependencies unless also directly referenced by your code; usually
more relevant on Linux with optional deps of system libraries or
Windows where you want to ignore many system DLLs)

--Ben