Defining ABI compatibility for a Go shared library

[Michael Hudson-Doyle]

Hi all,

I have a CL in review (https://golang.org/cl/8773) which aims to catch
ABI breaks during runtime initialization.

This is not really so much aimed at people invoking the Go tool for
themselves but more when packages are installed by distribution
packages. If the packager slips up and creates a version of a package
that contains a shared library that can be co-installed with a package
that was built against a version of that shared library with an older
ABI, we want to find out in a loud and obvious way, not by falling
apart at some random point.

So how do we define the ABI of a shared library? My branch:

1) Defines the ABI hash of a package as the SHA1 of the __.PKGDEF for
the package.
2) Defines the ABI hash of a shared library by sorting the contained
packages sort the packages by import path, concatenating the hashes of
the packages and computing the SHA1 of that

This falls on the strict side of the fence -- there are definitely
changes to a package one can make that do not break ABI in any
reasonable sense but will change this, e.g. changing the name of the
argument to an exported function. But that's the better side of the
fence to be on: I am pretty confident that a change that does not
change this file does not break ABI (because it's all the compiler
looks at when compiling a package that imports another). It also
includes the toolchain version, which is also required. Even the
build-id field that golang.org/cl//9154/ proposes adding is compatible
with this, although that could change (if e.g. the build-id included a
hash of the contents of the source files, we'd need to exclude it from
the abi hash calculation).

But it seems worth asking here: does this sound reasonable? Can you
think of any holes in the above or better ideas?

Cheers,
mwh

[minux]

We definitely need to take the compiler version and runtime version into

consideration (Go ABI is not externally defined, and it can change from

one compiler version to the next.)

What about use the hash of all the source (original source files only, but

also include assembly files), along with the checksum for all the packages

directly imported as the abi checksum for a package?

The benefit of this scheme is that if we make runtime depends on the

compiler packages, the checksum of any package will depend on both

the runtime and compiler version implicitly.

I think it's better to be more strict at this point than sorry.

[Russ Cox]

Michael's plan sounds right to me.

It's overkill to use the exact source content. One of the goals of .so files from the view of a Linux distributor is to be able to make security fixes by distributing just one updated .so file. If that can be done without affecting the export data corresponding to the .so, clients of that .so need not be recompiled and should not get spurious startup time version failures.

Put another way, the export data is the only part the compiler reads when importing a package. If the object file changes in a way that the export data stays the same, then the compiler would behave identically, since the part it reads is unchanged. The clients of that package need not be recompiled, just relinked. Therefore hashing the export data is just about exactly the right approximation.

If distributors wanted more flexibility later, the thing to do would be to arrange for the compiler to omit various optimiaztion information (like function bodies for inlining, and escape analysis results) from the export data, so that more could be changed without triggering the compatibility check. The actual check would stay the same.

I agree that the toolchain version is important too, but the toolchain version is in the export data.

Russ