Using go tool with third-party Go implementations

[Alexander Zhirov]

Hi all,

At the moment there are at least three alternative implementations of Go that I know of (and there are probably some I don't): gccgo, tinygo and gopherjs. Of the three, only gccgo is supported by the go tool out of box. Looking at the source code, gc ang gccgo have their own implementations of the toolchain interface.

The other two essentially reimplement the built system using go/build package, and, based on my experience with GopherJS, keeping up with changes to the upstream Go build tooling (modules, file embedding, etc.) is a lot of effort.

Out of interest, I was investigating what would it take for GopherJS to delegate build system details to the Go tool and only provide alternative implementations of compile/link tools. For prototyping purposes -toolexec flag allows me to insert a proxy that intercepts compiler/linker invocations and invokes GopherJS as required. The nice thing is that no GopherJS-specific code needs to exist in the cmd/go codebase.

 I'm currently at a very early stage, but it seems to be working out. Before I spend too much time though, I'd like to probe people's opinions of a few questions:

1) Is the idea of treating cmd/compile, cmd/link (and so on) interfaces as an extension point a reasonable one? I expect that they are going to change between releases, but probably fewer than the build system as a whole.

2) Using toolexec to intercept the calls works, but I assume this isn't it's intended purpose. I see no problem with using it, maybe I'm missing something?

3) Sort of related to #2, go tool would still check for supported GOOS/GOARCH pair, which is irrelevant to third-party compilers.

Cheers,

Alex.

[Ian Lance Taylor]

On Mon, Jan 11, 2021 at 11:03 AM Alexander Zhirov <al...@nevkontakte.com> wrote:
>
> At the moment there are at least three alternative implementations of Go that I know of (and there are probably some I don't): gccgo, tinygo and gopherjs. Of the three, only gccgo is supported by the go tool out of box. Looking at the source code, gc ang gccgo have their own implementations of the toolchain interface.
>
> The other two essentially reimplement the built system using go/build package, and, based on my experience with GopherJS, keeping up with changes to the upstream Go build tooling (modules, file embedding, etc.) is a lot of effort.
>
> Out of interest, I was investigating what would it take for GopherJS to delegate build system details to the Go tool and only provide alternative implementations of compile/link tools. For prototyping purposes -toolexec flag allows me to insert a proxy that intercepts compiler/linker invocations and invokes GopherJS as required. The nice thing is that no GopherJS-specific code needs to exist in the cmd/go codebase.
>
> I'm currently at a very early stage, but it seems to be working out. Before I spend too much time though, I'd like to probe people's opinions of a few questions:
>
> 1) Is the idea of treating cmd/compile, cmd/link (and so on) interfaces as an extension point a reasonable one? I expect that they are going to change between releases, but probably fewer than the build system as a whole.

I suspect this will give you trouble over time. When the go tool
invokes cmd/compile and cmd/link, it expects them to behave just like
the ordinary cmd/compile and cmd/link commands. For example, it
expects cmd/link to create a build ID that can be recognized by
cmd/internal/buildid. It expects to be able to pack additional
objects into archive files (see packInternal in
cmd/go/internal/work/gc.go). There may be other complex dependencies,
and certainly more will be added over time.

Ian

[Alexander Zhirov]

  > I suspect this will give you trouble over time.

That's the question I've been pondering a lot lately. I know for sure that trying to keep up with the go tool evolution is a lot of trouble. I'm trying to experiment with alternatives to get some sense for how they actually compare. Using go/build is what GopherJS currently does and it lacks feature parity with the go tool (which I think is intentional?). go/packages is a bit better and works reasonably for code analysis/editor support purposes, but doesn't lend itself for third-party compilers, and things like file embedding would still most likely require rebuilding from scratch (not 100% sure here, I only briefly skimmed the proposal at this point). Hijacking toolchain calls is the last option I haven't explored and in theory it sidesteps pretty much go tool-level problems.

> For example, it expects cmd/link to create a build ID that can be recognized by cmd/internal/buildid.  It expects to be able to pack additional objects into archive files (see packInternal in cmd/go/internal/work/gc.go).

This is a useful data point. Currently GopherJS uses its own bespoke format for intermediate artifacts, maybe it's one more reason to actually switch to proper object files. It works for WebAssembly, so we could probably make it work for JS as well. Is it worth the trouble? Maybe not, one way to find out :)

Another way to approach this is to implement a generic "third-party" toolchain with a simplified set of assumptions for all other compilers to use. From that perspective posing as gccgo would've been ideal if not for 1) gcc flags are a pain to parse in Go and 2) it doesn't produce calls to build standard library, since gccgo has it pre-built (as far as I understand).

пн, 11 янв. 2021 г. в 19:14, Ian Lance Taylor <ia...@golang.org>:

[Alexander Zhirov]

Thanks to Ian's pointers I was able to get to a state where "go build" command passes successfully. For the moment my compile/link/asm stubs produce minimally viable outputs (which are mostly empty), but the go tool is happy enough with the contents. Next step of plugging GopherJS logic in should not be actually too difficult. The good thing is, I only had to make a couple tiny patches to the go tool to make it all work:

1. Add GOOS=js GOARCH=js to the list of known pairs so that go tool doesn't refuse to work with me right away.
2. Add my custom signature to the list of known "object file" formats the tool should not be afraid to overwrite.

The diff is attached to the email, in case anyone's interested. I don't think that it is clean enough to be upstreamed in this exact form, but it proves that changes to enable this use case on the Go side would be minimal.

I've documented progress of this experiment so far in https://github.com/nevkontakte/gopherjs-plus/blob/tools/cmd/gopherjs-ng/README.md, and plan to keep hacking on it as time permits. 

пн, 11 янв. 2021 г. в 21:40, Alexander Zhirov <al...@nevkontakte.com>: