Half-baked proposal: pre-generate Cgo processing

[Elias Naur]

Inspired by https://github.com/golang/go/issues/35721#issuecomment-556427116

and my own desire for easily cross-compiling gioui.org programs, I wonder whether it is

feasable to re-use cgo processing through pre-generation of the cgo tools output.

Packages that import "C" require quite a lot of support software:

a native toolchain, development libraries, headers. Invoking the host compiler

is also slow. For example, cross-compiling Cgo programs for macOS and iOS is particularly

difficult (and perhaps legally impossible), and cross-compiling for Android requires

a ~700MB NDK.

One option is .syso files (https://github.com/golang/go/wiki/GcToolchainTricks),

pre-compiled native code that the Go linker knows how to link in. However .syso

files are too low level: they can't link to shared (system) libraries and there is no

Go<->C bridging for making calls across the language barrier safe.

So I wonder: can you imagine a Goldilocks option where a package can have access

all Cgo features through pre-generated cgo files? It's similar to the old binary-only package

feature, but only for the Cgo parts. I hope that C's much more stable ABI makes a pre-generated

cgo file format viable than pre-built Go binary packages.

I wouldn't mind if the hypothetical cgo file format changes once in a while, say every Go

release, as long as I could supply multiple files covering the Go releases I support. If none

of the pre-generated files are usable, the Go toolchain falls back to just running cgo processing.

In summary, I'd like to have a way to invoke

    $ go tool cgo build <package>

which then ouputs, say, <package_go115>.cgo. If <package> is later built, possibly on a different

machine, the Go toolchain will look for *.cgo files in the package directory and use one of them

if possible, skipping cgo processing altogether.

-- elias

[Aram Hăvărneanu]

On Solaris, Windows, and macOS we use cgo for system calls, and the
cgo processing is already pre-generated, as we can't assume any target
C compiler or linker. You can do the same for any C library, and I
have occasionally done it for one-off things, but it's quite annoying.

--
Aram Hăvărneanu

[Florian Weimer]

* Elias Naur:

> Inspired by
> https://github.com/golang/go/issues/35721#issuecomment-556427116 and
> my own desire for easily cross-compiling gioui.org programs, I wonder
> whether it is feasable to re-use cgo processing through pre-generation
> of the cgo tools output.

It doesn't work on GNU/Linux because the headers must match the
link-time library. The ABI is only frozen by the link editor. Before
that point, there is no ABI stability as far as glibc symbols are
concerned.

Thanks,
Florian

[Elias Naur]

Can the result of the link editor be recorded then? Go supports internal
linking Cgo on (some) Linuxes, right?

-- elias

[Florian Weimer]

* Elias Naur:

It requires recording the default symbol versions, and storing the code
from static libraries (libc_nonshared.a etc.). As far as I know, the
ELF format with GNU extensions allows to capture this information only
for programs, not for dynamic shared objects (i.e., if the Go code
itself goes into a DSO). But if the internal linker is used, a custom
storage format could be used.

Historically, there has been a tendency for the *_nonshared.a libraries
to shrink, but I do not know if they will go away completely. Some
glibc functions are required to be caller-sensitive, and putting them
into libc_nonshared.a would allow efficient caching of the identity of
the caller.

Thanks,
Florian

[Aram Hăvărneanu]

> It doesn't work on GNU/Linux because the headers must match the
> link-time library.

With internal linking, they do. There's no link-time library at
all, rather cgo generates code that uses //go:cgo_import_dynamic,
which has the same effect as using an import library (common in PE
world, but rare in ELF).

With external linking, we could still solve the relocations in the
internal linker when generating the Go object, although that would
defeat many of the advantages of external linking. A better solution
would be to generate a proper import library at cgo-generation time,
than the external linker can understand.

However, OP is concerned about the case where he doesn't have a
cross-toolchain installed, so only the internal-linking case is
applicable to him.

--
Aram Hăvărneanu

[Aram Hăvărneanu]

> Can the result of the link editor be recorded then? Go supports internal
> linking Cgo on (some) Linuxes, right?

The details are quite different, but this is essentially how cgo
works with internal linking. The cgo mechanism figures out the
symbols and generates (among other things) code that uses
//go:cgo_import_dynamic, which is later understood by the compiler
and the internal linker.

In the normal case, this code is thrown away, it's just some
intermediary step. But we could keep these files around (and we do,
for Solaris/Windows/macOS, though in that case the files are generated
by a different mechanism than in normal cgo mode).

I omitted quite a lot of details, especially around static linking.
There are some complications around static linking. If the target
library requires some static objects to be linked in, that causes
more problems.

--
Aram Hăvărneanu

[Aram Hăvărneanu]

> But if the internal linker is used, a custom storage format could
> be used.

This is what our cgo pragma directives do.

> and storing the code from static libraries (libc_nonshared.a etc.)

We could do this with .syso files, although it would be far from
ideal. Is libc_nonshared.a always used?

--
Aram Hăvărneanu

[Florian Weimer]

* Aram Hăvărneanu:

>> But if the internal linker is used, a custom storage format could
>> be used.
>
> This is what our cgo pragma directives do.

I see it now. Thanks, this is quite interesting.

>> and storing the code from static libraries (libc_nonshared.a etc.)
>
> We could do this with .syso files, although it would be far from
> ideal. Is libc_nonshared.a always used?

It is required for some functions: atexit, at_quick_exit,
pthread_at_fork, the stat family of functions (stat, fstat, lstat),
mknod and mknodat. For the stat/mknod functions, that is just
historical baggage without any current technical requirement. This may
get cleaned up during the Y2038 work.

Another piece of historical baggage is related to ELF constructors for
the main program. The statically linked C startup code passes the
address of functions in libc_nonshared.a to __libc_start_main in
libc.so.6. There is no technical requirement for that, either, and I
even posted patches once to fix this, but it requires changes to the
startup code for all targets, so it was … difficult to review.

In the future, it is possible that we will use a similar mechanism for
dlopen, dlmopen, dlsym, dlvsym, and perhaps another symbol lookup
function whose name is not yet known.

Thanks,
Florian