Go 2 suggestion - what dependencies included in a build?

[Eric Johnson]

As I scan reports of vulnerable software, I'm concerned that it is impossible to tell, from a Go binary, what was used to build that binary. Which means that if I depend on some library that is discovered to have a vulnerability, I cannot look at each of the binaries I have deployed, and discover if those binaries are vulnerable. This is starting to worry me, as my company builds more and more software with Go.

Now that "dep" is emerging as a standard tool, perhaps we can include the dependency information in the built binaries in a way that is discoverable with a common tool?

Possibly we also want to establish a convention, such as a command line parameter "--buildinfo" that can be specified to spit out that very info?

I'm happy to contributing to further exploring implementation, but I figured I'd start by asking a question, in case someone is already working in this direction.

Eric.

[Jan Mercl]

On Wed, Aug 16, 2017 at 1:58 AM 'Eric Johnson' via golang-nuts <golan...@googlegroups.com> wrote:

> I'm happy to contributing to further exploring implementation, but I figured I'd start by asking a question, in case someone is already working in this direction.

That'd be a nice tool for some to have and it's easy to implement it outside the Go tool chain.

--

-j

[Lars Seipel]

On Tue, Aug 15, 2017 at 04:58:00PM -0700, 'Eric Johnson' via golang-nuts wrote:
> As I scan reports of vulnerable software, I'm concerned that it is
> impossible to tell, from a Go binary, what was used to build that binary.

A lot of projects are already doing this, if somewhat indirectly: they
put a git commit hash into the binary that includes their (in-repo)
vendor directory.

Should vendor fall out of fashion at some point, one can surely come up
with a different scheme. I don't see why this should be tied to a
possible Go 2, though. It's pretty much independent from the language.

Might be more appropriate to work on it in the context of dep. Could be
as simple as making sure that dep can emit the required information for
people to pick up during their build.

[Eric Johnson]

I note it as something for Go 2, if only because it would be good to standardize it across all Go binaries, so it was possible to introspect every Go executable. Otherwise, I have to push to get all teams using go to adopt the same approach to building in this information, rather than having each team do it different. On top of that, it still doesn't solve my problem for third-party already-built binaries.

An indirect solution that captures the hash of the vendor directory (presumably the git commit hash?), doesn't solve the problem particularly well, even for internally built binaries, because I'd still then need to go back to the source tree to answer the question - "is this executable vulnerable?" That's more time consuming, more prone to failure, and the failure modes are harder to resolve (don't have access to source code, network is down, etc.).

We don't need to worry about this with dynamically linked libraries, because we can actually see the versions of the libraries on the system directly. To make the Go binaries functionality equivalent, I want to be able to see the answer to the question without having to go to separate systems.

Eric.

[Lucio]

On Wednesday, 16 August 2017 01:58:00 UTC+2, Eric Johnson wrote:

[ ... ]

I'm happy to contributing to further exploring implementation, but I figured I'd start by asking a question, in case someone is already working in this direction.

I was thinking that it ought to be possible to re-arrange modules in a package quite dramatically by "optimising" the import list. To be clearer, once the compiler or a similar tool has a memory image that contains all the imported modules, it ought to be able (with much effort, I allow) to redefine the module boundaries to match the import list: functions (and hopefully variables) dependent on different sets of imported packages would be relocated to distinct (output) modules.

I hope my lack of theoretical knowledge does not show too clearly in the above :-).

Lucio.

[Jakob Borg]

Keep in mind that you can't assume vcs info is available at build time. They may be building from a downloaded tarball, in which case you *may* have a Gopkg.lock (if everyone uses dep) but not much else. They may be Debian and build from source packages where the Go compiler sees no version info at all - but the package manager knows the information you want. Etc. 

//jb

[Eric Johnson]

The compiler, at least, knows which packages were used to compile the source. Even absent additional metadata from something like Gopkg.lock, the compiler could still include info about the packages compiled into a binary. Knowing that something might be vulnerable - but not knowing the version from the executable - is still better than not knowing at all.

Eric.

[Lars Seipel]

On Thu, Aug 17, 2017 at 09:54:43AM -0700, Eric Johnson wrote:
> The compiler, at least, knows which packages were used to compile the
> source. Even absent additional metadata from something like Gopkg.lock, the
> compiler could still include info about the packages compiled into a
> binary. Knowing that something might be vulnerable - but not knowing the
> version from the executable - is still better than not knowing at all.

If the set of packages as described by their import paths is useful to
you, you already get that today by calling nm (or 'go tool nm') on the
binary.

The printed list of symbols should give you a pretty good idea about the
contents of a particular binary, just without explicit version
information.