build process to require Go 1.4 for bootstrap

[Russ Cox]

It's time to start having parts of the Go toolchain written in Go,
which means we need to figure out how the bootstrap will work. I
propose that we require having Go 1.4 available in order to build
future versions of Go.

Details at golang.org/s/go15bootstrap. Comments welcome in this thread.

Thanks.
Russ

[Aram Hăvărneanu]

Are there plans for an automagic bootstrap.sh that builds Go 1.4 and
then builds Go 1.x with it? (I'm ambivalent, just asking for
completeness).

Will it be possible to boostrap the gc implementation of Go 1.x with
gccgo? Related to this question, why is $GOROOT_BOOTSTRAP a directory
rather than a path to a 1.4 go tool?

--
Aram Hăvărneanu

[Russ Cox]

On Wed, Jan 7, 2015 at 11:58 AM, Aram Hăvărneanu <ara...@mgk.ro> wrote:
> Are there plans for an automagic bootstrap.sh that builds Go 1.4 and
> then builds Go 1.x with it? (I'm ambivalent, just asking for
> completeness).

I do not plan to do that, at least not until there is a demonstrated need.

> Will it be possible to boostrap the gc implementation of Go 1.x with
> gccgo?

In principle, that should work. It would require setting
$GOROOT_BOOTSTRAP to point to a GOROOT tree where bin/go can be run
and invokes gccgo by default. I don't know enough about gccgo to say
how you do that, but if you have gccgo running on the machine,
probably you already know how.

> Related to this question, why is $GOROOT_BOOTSTRAP a directory
> rather than a path to a 1.4 go tool?

It is a directory because that tells us which $GOROOT to set when invoking it.
If you unpack a binary Go 1.4 release into $HOME/go1.4 it won't work
without $GOROOT set.

Russ

[Josh Bleecher Snyder]

> Details at golang.org/s/go15bootstrap. Comments welcome in this thread.

Throughout the doc, should we read 1.4 as 1.4.x?

-josh

[Ian Lance Taylor]

Yes, the build procedure for 1.4 is not going to change, and it should
work to start the build with any 1.4 release (unless we find some
fatal bug in 1.4, which seems unlikely).

Ian

[Dave Cheney]

What changes will be needed to the builders ? Should dashboard/builder
fetch and build a version of 1.4 for GOROOT_BOOTSTRAP, or should it
just expect that to be present already as a prerequisite ?

I don't have a preference, the latter is more straight forward, but
the former shouldn't be that hard to add either.

> --
> You received this message because you are subscribed to the Google Groups "golang-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Dave Cheney]

> I don't have a preference, the latter is more straight forward, but
> the former shouldn't be that hard to add either.

Actually I do have a preference. On reflection I think the builders
should expect $GOROOT_BOOTSTRAP to be present and populated before
starting up.

[Russ Cox]

That's what I was envisioning, but I am happy to leave that decision to people who actually know about the current state of builders.

Russ

 

[Brad Fitzpatrick]

For the old-style builders, having $GOROOT_BOOTSTRAP already sitting there is fine. If things go well, we won't have many of those left.

For the new-style builders, we push files to the machines and tell them to run commands. The root directory that we push into is decided by the VMs themselves (some temp directory) and the build coordinator doesn't now the absolute paths. I figure we'd just push in Go 1.4 in the right spot.

My concern then is the Go 1.4 we push may not know its GOROOT, and if we set $GOROOT in the environment, which world does that affect? $GOROOT_BOOTSTRAP says where to find Go 1.4, but $GOROOT_BOOTSTRAP/bin/go has been potentially relocated from when it was first built so needs a real $GOROOT variable to tell it where to finds its source & packages. But does setting setting both $GOROOT and $GOROOT_BOOTSTRAP to $TMP/go1.4 cause the built Go 1.5 to get messed up?

[minux]

Yeah, I still like the idea of passing a $GO_BOOTSTRAP that points

to the go command from Go 1.4, rather than pass a the GOROOT

in $GOROOT_BOOTSTRAP

For binary distributions, just install to /usr/local/go, or write a wrapper

script.

I don't install Go 1.4 at $HOME/go1.4, but I do have go1.4 in my $PATH,

so if the default value for $GO_BOOTSTRAP is go1.4, then the procedure

will work without forcing people to create a $HOME/go1.4 (to save the

trouble setting GOROOT_BOOTSTRAP.)

[Ian Lance Taylor]

On Wed, Jan 7, 2015 at 5:18 PM, Brad Fitzpatrick <brad...@golang.org> wrote:
>
> My concern then is the Go 1.4 we push may not know its GOROOT, and if we set
> $GOROOT in the environment, which world does that affect? $GOROOT_BOOTSTRAP
> says where to find Go 1.4, but $GOROOT_BOOTSTRAP/bin/go has been potentially
> relocated from when it was first built so needs a real $GOROOT variable to
> tell it where to finds its source & packages. But does setting setting both
> $GOROOT and $GOROOT_BOOTSTRAP to $TMP/go1.4 cause the built Go 1.5 to get
> messed up?

I suppose we need GOROOT_BOOTSTRAP_GOROOT.

Or, like, a shell script or something.

Ian

[Brad Fitzpatrick]

Yeah, I think a shell script will be the answer. So no objections on this proposal from me... I'll make it work.

[Andrew Gerrand]

Yep SGTM

[minux]

Given that the builders need a script anyway, why not use $GO_BOOTSTRAP that

points to the Go 1.4 go command rather than $GOROOT_BOOTSTRAP?

[Rob Pike]

I suggest $GO_BIKESHED.

-rob

[Brad Fitzpatrick]

You can own the builders if you'd like.

[minux]

Do we still need the cmd/dist tool after everything is in Go?

if we require that we specify every part of the toolchain uses only Go 1.4,

I imagine, we just need to use a couple go build commands to build the

toolchain, put it in place, and set GOROOT to the current GOROOT, and

use the bootstrap go command to build the go_bootstrap command.

(this step is tricky, we essentially make the Go command from Go 1.4

operate in a Go 1.5 GOROOT, but I don't think that's impossible: as long

as the toolchain command line are compatible with Go 1.4's, but this is

easy, as current toolchain command lines are already compatible with

Go flag package.)

And once go_bootstrap is available, use it to rebuild everything (including

the toolchain) with Go 1.5 packages.

Given that most files that used to be generated by cmd/dist are now

generated statically using go:generate, I imagine we can get rid of

cmd/dist once everything is in Go.

The other job of cmd/dist: detecting GOARCH/GOHOSTARCH, etc,

are not needed because we're requiring a working Go 1.4 installation,

which contains these information already.

Cross compiling users need to set GOOS and GOARCH explicitly,

and that is true today.

[Brad Fitzpatrick]

I forgot to punctuate all sentences with emoticons.

[Rob Pike]

💩 😄

-rob

[Volker Dobler]

Details at golang.org/s/go15bootstrap. Comments welcome in this thread.

Bootstrapping gcc has one additional step where convergence

of the generated compiler toolchain to a fixed point is checked.

Let Go1.x_{Go1.y} be the Go compiler toolchain 1.x built with

Go 1.y. With this notation step 2 in Russ' proposal produces

Go1.x_{Go1.4} and step 3 produces Go1.x_{Go1.x_{Go1.4}} which

is used to build cmd/go as go_bootstrap in the proposal.

Would it be sensible to add one more step 3.5 which builds

the Go1.x_{Go1.x_{Go1.x_{Go1.4}}} toolchain and verifies it is

"the same" as Go1.x_{Go1.x_{Go1.4}}?

V.

[Matthew Dempsky]

On Wed, Jan 7, 2015 at 8:06 AM, Russ Cox <r...@golang.org> wrote:

Details at golang.org/s/go15bootstrap. Comments welcome in this thread.

Two minor comments:

In the case of building a cross-compiler toolchain, does "build Go 1.x compiler toolchain" mean building the full toolchain capable of targeting both $GOHOSTOS/$GOHOSTARCH and $GOOS/$GOARCH?  If so, one minor optimization is the step #2 toolchain could omit support for $GOOS/$GOARCH as long as step #3 will follow; but if step #3 is omitted (e.g., for debugging, as described in the doc), the step #2 toolchain would need to support the target platform.

Suppose we add support for openvms/vax to Go 1.6.  How would the release binaries be built?  Would they be need to be "Canadian cross-compiled" from an existing Go 1.4 platform?  Would Go 1.7 development on openvms/vax continue to be officially developed this way, or would it bootstrap natively from Go 1.6?  If openvms/vax starts bootstrapping from Go 1.6, should the rest of the platforms update to Go 1.6 for consistency?  Admittedly, answers aren't really needed until support for a new platform shows up, but it seems worth considering to make sure there are acceptable options.

[Ian Lance Taylor]

On Thu, Jan 8, 2015 at 3:35 PM, 'Matthew Dempsky' via golang-dev
<golan...@googlegroups.com> wrote:
>
> Suppose we add support for openvms/vax to Go 1.6. How would the release
> binaries be built? Would they be need to be "Canadian cross-compiled" from
> an existing Go 1.4 platform?

The Go team only builds release binaries for a subset of existing
platforms: GNU/Linux, Darwin, Windows, FreeBSD. In general we're only
going to build release binaries for systems we actually have. So I
don't think that building release binaries is an issue.

> Would Go 1.7 development on openvms/vax
> continue to be officially developed this way, or would it bootstrap natively
> from Go 1.6?

I imagine most developers would bootstrap natively from 1.6, setting
BOOTSTRAP_GOROOT appropriately. That is, there is nothing requiring
BOOTSTRAP_GOROOT to be 1.4; that is the default, and we need to
promise that 1.4 will work so that people can bootstrap up, but
everything will work if BOOTSTRAP_GOROOT is some version later than
1.4.

> If openvms/vax starts bootstrapping from Go 1.6, should the
> rest of the platforms update to Go 1.6 for consistency?

No reason for that. Since Go is strictly backward compatible, the
fact that the other builders will start from 1.4 means that
openvms/vax, starting from 1.6, will continue to work reliably.

Ian

[Matthew Dempsky]

On Thu, Jan 8, 2015 at 4:13 PM, Ian Lance Taylor <ia...@golang.org> wrote:

On Thu, Jan 8, 2015 at 3:35 PM, 'Matthew Dempsky' via golang-dev
<golan...@googlegroups.com> wrote:
> Suppose we add support for openvms/vax to Go 1.6.

(To be clear: openvms/vax was a jestful stand-in for $FUTUREOS/$FUTUREARCH, and not a platform I anticipate anyone will actually add support for, much less that Go team would care to officially support.)

> How would the release
> binaries be built?  Would they be need to be "Canadian cross-compiled" from
> an existing Go 1.4 platform?

The Go team only builds release binaries for a subset of existing
platforms: GNU/Linux, Darwin, Windows, FreeBSD.  In general we're only
going to build release binaries for systems we actually have.  So I
don't think that building release binaries is an issue.

Understood that Go team only builds release binaries for the subset of platforms that they're most interested in.  I just expect that set is likely to change over time, which means Go team will someday need to tackle the release bootstrapping problem somehow.

No reason for that.  Since Go is strictly backward compatible, the
fact that the other builders will start from 1.4 means that
openvms/vax, starting from 1.6, will continue to work reliably.

That's what I would hope at least.  I'm just curious if there could be any subtleties that might arise.

E.g., in OpenBSD, a couple of our older CPU architectures are stuck using GCC 3.3 still, whereas most instead use GCC 4.2.  GCC 4.2 is backwards compatible with GCC 3.3, but developers occasionally forget GCC 3.3's nuances and end up committing code that breaks on those older architectures, and which doesn't get detected until much later.

Thankfully there are a number of reasons why this particular example shouldn't affect the Go project.  I just present it as an example of the class of toolchain-desync issues I had in mind.

[Russ Cox]

No, it all just works. GOROOT is set to GOROOT_BOOTSTRAP when invoking Go 1.4 to build something and set to the usual (different) value for Go 1.5. There is no need for a script, just a Go 1.4 checkout in the GOROOT_BOOTSTRAP directory and a tiny amount of care taken to set GOROOT appropriately at different times during the build. It doesn't matter what the baked-in GOROOT is in the Go 1.4 checkout. The whole point of GOROOT_BOOTSTRAP being a directory is to avoid needing a script.

I just did

mv $HOME/go1.4 $HOME/go-somewhere-else

GOROOT_BOOTSTRAP=$HOME/go-somewhere-else ./all.bash

and it worked fine (with https://go-review.googlesource.com/2470).

Russ

 

[Brad Fitzpatrick]

Good to hear. That makes things a bit easier than I was expecting.

[Russ Cox]

On Wed, Jan 7, 2015 at 9:26 PM, minux <mi...@golang.org> wrote:

Do we still need the cmd/dist tool after everything is in Go?

Yes. Something has to manage the build. In fact, once cmd/dist is written in Go I am going to move the bottom half of make.bash/make.bat/make.rc and all of run.bash/run.bat/run.rc into cmd/dist. It will be far better to maintain one copy written in Go than three copies written in different scripting languages. run.bash will become a 2-line script:

    #!/bin/bash

    exec go tool dist run "$@"

I thought about this for a long time before writing the doc. It may be possible to get something like this to work, but that couples the build to details of an earlier Go release more tightly than I would like. The very significant advantage of cmd/dist is that it contains all the rules for building the actual Go 1.5 toolchain. You'll note that even in the converted Go version I am not using the go/build package. That is deliberate. cmd/dist will be in sync with the release being built. Reusing too much of the build mechanism from an earlier Go release will make it too easy to be out of sync. 

cmd/dist is a tiny program and it buys us a lot in flexibility. It is well worth the cost.

You are correct that there is (at least today) no longer any need to compute the setting for GOHOSTOS. It gets "learned" in very much the same way that Ken's Trusting Trust talk explained how the C compiler learns \v. There is still a need to detect an appropriate GOHOSTARCH and GOARCH, however. For example if you have a 386 Go 1.4 toolchain on a 64-bit machine, you should still be able to build an amd64 Go 1.5 toolchain. You can see what stayed and didn't in the converted cmd/dist (https://go-review.googlesource.com/2470).

Russ

[Russ Cox]

I'm going to stop at Go1.x_{Go1.x_{Go1.4}} until someone demonstrates that we must go deeper.

I don't mean to brush off your question. I really don't know, and it doesn't seem worth worrying about until a problem arises.

Russ

[Russ Cox]

On Thu, Jan 8, 2015 at 6:35 PM, Matthew Dempsky <mdem...@google.com> wrote:

On Wed, Jan 7, 2015 at 8:06 AM, Russ Cox <r...@golang.org> wrote:

Details at golang.org/s/go15bootstrap. Comments welcome in this thread.

Two minor comments:

In the case of building a cross-compiler toolchain, does "build Go 1.x compiler toolchain" mean building the full toolchain capable of targeting both $GOHOSTOS/$GOHOSTARCH and $GOOS/$GOARCH?  If so, one minor optimization is the step #2 toolchain could omit support for $GOOS/$GOARCH as long as step #3 will follow; but if step #3 is omitted (e.g., for debugging, as described in the doc), the step #2 toolchain would need to support the target platform.

Thanks. I believe you're right. It will be a while before we're far enough along to need that, though. :-)

Russ

[Russ Cox]

On Thu, Jan 8, 2015 at 7:42 PM, Matthew Dempsky <mdem...@google.com> wrote:

On Thu, Jan 8, 2015 at 4:13 PM, Ian Lance Taylor <ia...@golang.org> wrote:

On Thu, Jan 8, 2015 at 3:35 PM, 'Matthew Dempsky' via golang-dev
<golan...@googlegroups.com> wrote:
> Suppose we add support for openvms/vax to Go 1.6.

(To be clear: openvms/vax was a jestful stand-in for $FUTUREOS/$FUTUREARCH, and not a platform I anticipate anyone will actually add support for, much less that Go team would care to officially support.)

> How would the release
> binaries be built?  Would they be need to be "Canadian cross-compiled" from
> an existing Go 1.4 platform?

The Go team only builds release binaries for a subset of existing
platforms: GNU/Linux, Darwin, Windows, FreeBSD.  In general we're only
going to build release binaries for systems we actually have.  So I
don't think that building release binaries is an issue.

Understood that Go team only builds release binaries for the subset of platforms that they're most interested in.  I just expect that set is likely to change over time, which means Go team will someday need to tackle the release bootstrapping problem somehow.

As Ian said, in general we're only going to build release binaries for systems we actually have. If we're interested enough to build release binaries, our solution will be to get a (physical or virtual) machine on which to build them, not to complicate the build process. We'll need builders for the dashboard anyway.

Russ

[binaryc...@gmail.com]

On Wednesday, January 7, 2015 at 9:05:32 AM UTC-8, rsc wrote:

On Wed, Jan 7, 2015 at 11:58 AM, Aram Hăvărneanu <ara...@mgk.ro> wrote:
> Are there plans for an automagic bootstrap.sh that builds Go 1.4 and
> then builds Go 1.x with it? (I'm ambivalent, just asking for
> completeness).

I do not plan to do that, at least not until there is a demonstrated need.

> Will it be possible to boostrap the gc implementation of Go 1.x with
> gccgo?

In principle, that should work. It would require setting
$GOROOT_BOOTSTRAP to point to a GOROOT tree where bin/go can be run
and invokes gccgo by default. I don't know enough about gccgo to say
how you do that, but if you have gccgo running on the machine,
probably you already know how.

> Related to this question, why is $GOROOT_BOOTSTRAP a directory
> rather than a path to a 1.4 go tool?

It is a directory because that tells us which $GOROOT to set when invoking it.
If you unpack a binary Go 1.4 release into $HOME/go1.4 it won't work
without $GOROOT set.

(Not directed at Russ); I am particularly interested in this approach (using gccgo), so if anyone tries this and is successful, please respond back for the benefit of others!  (I plan to try this as soon as I can.)

-Shawn
 

[minux]

On Fri, Jan 9, 2015 at 2:09 PM, <binaryc...@gmail.com> wrote:

(Not directed at Russ); I am particularly interested in this approach (using gccgo), so if anyone tries this and is successful, please respond back for the benefit of others!  (I plan to try this as soon as I can.)

There isn't much Go code in the toolchain yet.

It really depends on the gccgo version, and whether the Go toolchain uses

Go 1.4 only feature (because gccgo trunk doesn't support full Go 1.4 yet)

What I'm more interested in is the ability to generate a C output from Go.

This is mentioned in the original design docs for the Go translation, so we

still have the ability to bring up Go on a new platform without native Go

toolchain installed and without resorting to cross compilation. (Use qemu

user mode emulation is also an interesting solution to bring up Go on

unsupported architectures.)

Or perhaps having PNaCl support is enough, if you enable the file system

accesses option.