darwin/arm64 bootstrap binary kit

[Benny Siegert]

Hi!

Is there a bootstrap binary kit for macOS on Apple Silicon available?
This has been raised as a feature request in the pkgsrc package
collection.

Thanks in advance!

--
Benny

[cherry]

The binary release is tracked at https://github.com/golang/go/issues/42756 . We may provide the binary in 1.16 beta sometime soon.

In the meantime, you can use "GOARCH=arm64 GOOS=darwin ./bootstrap.bash" to build a bootstrap toolchain.

Thanks,

Cherry

--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-dev/CAN%2BFjHPzF0EZ3qtywN23TQtxeTH7QJ000JdCVszvn1dfs8Mjug%40mail.gmail.com.

[jpap]

Does a manual build of go on Apple M1 require Rosetta?

I have a (self-built) arm64 bootstrap compiler based on 1.16beta1, where $GOROOT_BOOTSTRAP/bin/go is an arm64 binary.  When I use that with $GOROOT/src/make.bash, the resulting $GOROOT/bin/go binary is x86_64, and produces darwin/amd64 binaries by default.  (You need to use GOARCH=arm64 to produce darwin/arm64 binaries using it.)  Make.bash also installs a darwin/arm64 binary into $GOROOT/bin/darwin_arm64/go which produces darwin/arm64 binaries by default.  See the build log below.

I was expecting $GOROOT/bin/go to be an arm64 binary in this case, and for it to produce darwin/arm64 builds by default.  It looks like $GOROOT/src/make.bash assumes GOHOSTARCH=amd64...? (All commands were run in a terminal where `arch` reports arm64.)

I just downloaded the official unstable darwin/arm64 1.16beta1 build and it includes only one darwin/arm64 binary as $GOROOT/bin/go, which is what I was expecting above.  Is this download just a bootstrap build based on 1.16beta1?  How is it built differently to the above where you end up with an x86_64 go binary in addition to the darwin_arm64/go arm64 binary?

Thanks,

John

m1:src jpap ((go1.16beta1))$ GOROOT_BOOTSTRAP=$HOME/go GOARCH=arm64 GOOS=darwin ./bootstrap.bash

#### Copying to ../../go-darwin-arm64-bootstrap

#### Cleaning ../../go-darwin-arm64-bootstrap

Removing VERSION.cache

Removing bin/

Removing misc/git/

Removing pkg/darwin_386/

Removing pkg/darwin_amd64/

Removing pkg/darwin_arm64/

Removing pkg/include/

Removing pkg/obj/

Removing pkg/tool/

Removing src/cmd/cgo/zdefaultcc.go

Removing src/cmd/go/internal/cfg/zdefaultcc.go

Removing src/cmd/go/internal/cfg/zosarch.go

Removing src/cmd/go/testdata/bin/

Removing src/cmd/internal/objabi/zbootstrap.go

Removing src/go/build/zcgo.go

Removing src/runtime/internal/sys/zversion.go

#### Building ../../go-darwin-arm64-bootstrap

Building Go cmd/dist using $HOME/go. (go1.15.6 darwin/amd64)

Building Go toolchain1 using $HOME/go.

Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.

Building Go toolchain2 using go_bootstrap and Go toolchain1.

Building Go toolchain3 using go_bootstrap and Go toolchain2.

Building packages and commands for host, darwin/amd64.

Building packages and commands for target, darwin/arm64.

----

Bootstrap toolchain for darwin/arm64 installed in $HOME/go/src/github.com/golang/go-darwin-arm64-bootstrap.

Building tbz.

-rw-r--r--  1 jpap  staff  127958037 Dec 23 14:12 $HOME/go/src/github.com/golang/go-darwin-arm64-bootstrap.tbz

m1:src jpap ((go1.16beta1))$ file $HOME/go/src/github.com/golang/go-darwin-arm64-bootstrap/bin/go

$HOME/go/src/github.com/golang/go-darwin-arm64-bootstrap/bin/go: Mach-O 64-bit executable arm64

m1:src jpap ((go1.16beta1))$ GOROOT_BOOTSTRAP=$(pwd)/../../go-darwin-arm64-bootstrap GOARCH=arm64 GOOS=darwin ./make.bash

Building Go cmd/dist using $HOME/go/src/github.com/golang/go/src/../../go-darwin-arm64-bootstrap. (devel +2ff33f5e44 Thu Dec 17 16:03:19 2020 +0000 darwin/arm64)

Building Go toolchain1 using $HOME/go/src/github.com/golang/go/src/../../go-darwin-arm64-bootstrap.

Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.

Building Go toolchain2 using go_bootstrap and Go toolchain1.

Building Go toolchain3 using go_bootstrap and Go toolchain2.

Building packages and commands for host, darwin/amd64.

Building packages and commands for target, darwin/arm64.

---

Installed Go for darwin/arm64 in $HOME/go/src/github.com/golang/go

Installed commands in $HOME/go/src/github.com/golang/go/bin

m1:src jpap ((go1.16beta1))$ file $HOME/go/src/github.com/golang/go/bin/go

$HOME/go/src/github.com/golang/go/bin/go: Mach-O 64-bit executable x86_64

m1:src jpap ((go1.16beta1))$ file $HOME/go/src/github.com/golang/go/bin/darwin_arm64/go

$HOME/go/src/github.com/golang/go/bin/darwin_arm64/go: Mach-O 64-bit executable arm64

m1:src jpap ((go1.16beta1))$ 

[cherry]

On Wed, Dec 23, 2020 at 5:55 PM jpap <jpap...@gmail.com> wrote:

Does a manual build of go on Apple M1 require Rosetta?

I have a (self-built) arm64 bootstrap compiler based on 1.16beta1, where $GOROOT_BOOTSTRAP/bin/go is an arm64 binary.  When I use that with $GOROOT/src/make.bash, the resulting $GOROOT/bin/go binary is x86_64,

I have been doing this for quite a while, and it results in ARM64 binaries as expected. What is "uname -m" output on your machine?

 

[jpap]

m1:~ jpap$ uname -m

arm64

m1:~ jpap$ arch

arm64

m1:~ jpap$ 

I've noticed a similar log from FiloSottile, though that build appears to be using an amd64 bootstrap.

[cherry]

Hmmm, I cannot reproduce.

% GOROOT_BOOTSTRAP=$HOME/go-darwin-arm64-bootstrap ./make.bash
Building Go cmd/dist using /Users/cherry/go-darwin-arm64-bootstrap. (devel +ba515efe12 Fri Nov 27 01:08:51 2020 -0500 darwin/arm64)
Building Go toolchain1 using /Users/cherry/go-darwin-arm64-bootstrap.

Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.

Building packages and commands for darwin/arm64.
---
Installed Go for darwin/arm64 in /Users/cherry/src/go
Installed commands in /Users/cherry/src/go/bin
cherry@Cherrys-Mac src % file ../bin/go
../bin/go: Mach-O 64-bit executable arm64

On Wed, Dec 23, 2020 at 6:49 PM jpap <jpap...@gmail.com> wrote:

m1:~ jpap$ uname -m

arm64

m1:~ jpap$ arch

arm64

m1:~ jpap$ 

 

Well, I don't know, then. Maybe double check your environment. Did you set GOHOSTARCH?

 

I've noticed a similar log from FiloSottile, though that build appears to be using an amd64 bootstrap.

Yeah, if the bootstrap toolchain is AMD64, you'll get an AMD64 go binary in GOROOT/bin/go by default. (But not with an ARM64 bootstrap toolchain.)

 

On Wednesday, December 23, 2020 at 3:10:35 PM UTC-8 cherry wrote:

I have been doing this for quite a while, and it results in ARM64 binaries as expected. What is "uname -m" output on your machine?

 

--

You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-dev/7e8551f1-b75b-4c88-a9d3-00991a60d0c0n%40googlegroups.com.

[jpap]

Thank you for double-checking, Cherry.

After learning how the build is set up in `src/make.bash` and `src/cmd/dist/build.go`, I realized make.bash is being run under bash, while my shell uses the macOS-default zsh.  The command `/usr/bin/env bash -c "uname -m"` reported x86_64, which explains why the build was set up for cross-compilation (build uses bash), despite "uname -m" reporting arm64 earlier (my shell uses zsh).

It turns out I had an old homebrew install of x86_64 bash on the system, a carry-over by Migration Assistant from my old Intel Mac.

With the x86_64 bash gone, the arm64 go build runs as expected.

jpap

[Rob Pike]

I have this issue as well, and haven't figured out why. Unless I set GOHOSTARCH explicitly, make.bash builds host amd64 for target arm64. Both name -m and mach report arm64. I can't work out where the x86 default is coming from.

I also have to set CGO_ENABLED=0 because of an assembly language syntax error (something about [reg] notation) that stops the build.

I can go back and recover the exact errors, but the travails to get me to a working native build were sufficient to stop me wanting to reset. Push me if you want to see them.

I'm running the latest everything: Go source, OS, xcode tools all just updated.

-rob

--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-dev/f09ca137-296b-4fee-804d-2c1cfdb0d1ean%40googlegroups.com.

[Rob Pike]

s/name/uname/

s/mach/arch/

s/me/someone who can type/

[jpap]

Hey Rob,

On Wednesday, December 23, 2020 at 8:19:01 PM UTC-8 r...@golang.org wrote:

I have this issue as well, and haven't figured out why. Unless I set GOHOSTARCH explicitly, make.bash builds host amd64 for target arm64. Both name -m and mach report arm64. I can't work out where the x86 default is coming from.

I went down the same GOHOSTARCH path which can be solved with the right compiler flags (see below).  A better method is to just remove the x86_64 taint, which for me came from an old bash install carried over from homebrew and Migration Assistant from an Intel Mac.  Removing that fixed it for me.

The simplest way to ensure that you avoid Rosetta and the x86_64 taint is to:

1. Ensure you're running an arm64 bash or zsh shell (`uname -m` or `arch` reporting arm64 and not x86_64).

2. Run the `src/make.bash` script without invoking the script's interpreter directive, using `. ./make.bash` from the src directory.  This avoids the `make.bash` script invoking an x86_64 shell under Rosetta.

3. Use the official go1.16beta1 arm64 binary distribution as the bootstrap compiler. (Or build one manually.)

I also have to set CGO_ENABLED=0 because of an assembly language syntax error (something about [reg] notation) that stops the build.

You probably mean this,

    # runtime/cgo

    gcc_arm64.S:28:16: error: brackets expression not supported on this target

    stp x29, x30, [sp, #-96]!

                  ^

    gcc_arm64.S:32:2: error: unknown use of instruction mnemonic without a size suffix

    mov x29, sp

    ^

    gcc_arm64.S:34:16: error: brackets expression not supported on this target

    stp x19, x20, [sp, #80]

                  ^

    [snipped...]

which comes from the build attempting to compile src/runtime/cgo/gcc_arm64.S which uses the ARM syntax, but clang is being invoked from the x86_64 environment, so its default target is x86_64 and expects the Intel syntax.

You could force clang to target arm64 by giving it the right flags:

    CGO_LDFLAGS="-arch arm64 -isysroot `xcrun --sdk macosx11.1 --show-sdk-path`" \

    CGO_CFLAGS="-arch arm64 -isysroot `xcrun --sdk macosx11.1 --show-sdk-path`" \

    GOHOSTARCH=arm64 \

    . ./make.bash

... which will build an arm64 go toolchain from a Rosetta x86_64 shell.  (You might need a different macosx SDK version depending on which version of Big Sur you're on.)

jpap

[Rob Pike]

Thanks, that was very helpful.

The step that I was missing was using an arm installation as the bootstrap. As far as I know, the taint as you memorably call it is not documented, and I'm so comfortable with cross-builds that it surprises me to see it exist at all.

The machine is awesome, and I trust the Go installation process will become smoother. (I know, I'm building from source, what's wrong with me, but that's who I am.)

-rob

--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-dev/976e68ba-60a4-4b07-b0a7-fe1ddeed00b5n%40googlegroups.com.

[cherry]

You could force clang to target arm64 by giving it the right flags:

    CGO_LDFLAGS="-arch arm64 -isysroot `xcrun --sdk macosx11.1 --show-sdk-path`" \

    CGO_CFLAGS="-arch arm64 -isysroot `xcrun --sdk macosx11.1 --show-sdk-path`" \

As far as I can tell, the C compiler on macOS/ARM64 emits ARM64 binaries by default if the compiler is invoked as an ARM64 binary, and emits AMD64 binaries if the compiler is invoked as an AMD64 binary. The flag is not necessary if it is invoked as an ARM64 binary. However, it is unclear to me how the system decides which architecture to run for the C compiler (or a universal binary in general). Apparently, the parent process' architecture seems to matter. The parent's parent (and more ancestors) also?

Is the -isysroot necessary? I don't think it is necessary if you are building on an M1 (even as a different architecture).

Maybe it would be helpful to ensure that the build is performed on an ARM64 environment and prevent AMD64 from "slipping in". This includes

- using an ARM64 bootstrap toolchain (otherwise generate one using bootstrap.bash)

- using an ARM64 bash. Maybe run make.bash with "arch -arm64 /bin/bash ./make.bash" ?

Is there anything else?

Thanks!

 

[cherry]

If you're running on an x86 shell and using an ARM64 go, "arch -arm64 go build" should work, to let the C compiler do the right thing.

[jpap]

On Mon, Dec 28, 2020 at 11:00 AM cherry <luna...@gmail.com> wrote:

You could force clang to target arm64 by giving it the right flags:

    CGO_LDFLAGS="-arch arm64 -isysroot `xcrun --sdk macosx11.1 --show-sdk-path`" \

    CGO_CFLAGS="-arch arm64 -isysroot `xcrun --sdk macosx11.1 --show-sdk-path`" \

As far as I can tell, the C compiler on macOS/ARM64 emits ARM64 binaries by default if the compiler is invoked as an ARM64 binary, and emits AMD64 binaries if the compiler is invoked as an AMD64 binary. The flag is not necessary if it is invoked as an ARM64 binary.

That's right: clang invoked under X \in {arm64, x86_64} builds for X unless the -arch flag is specified.

It's unfortunate that your quote removed the context here: if the build fails with weird [reg] notation, like what Rob saw, it's likely there's something on the system that makes clang run under x86_64 so it targets intel unless instructed otherwise.  The CGO_* flags instruct clang to target arm64, even if it still runs under x86_64, which fixes the build.

In the following example run on M1, my shell is the default macOS zsh (arm64), bash is x86_64 (via homebrew which carried over from Migration Assistant), and I am using a darwin/amd64 go bootstrap.  I've edited src/make.bash to print the current arch which is shown in bold below.

  $ echo $SHELL

  /bin/zsh

  $ pwd

  /Users/jpap/go/src

  $ git branch

  * (HEAD detached at go1.16beta1)

  $ arch

  arm64

  $ CGO_LDFLAGS="-arch arm64" \

    CGO_CFLAGS="-arch arm64" \

    GOHOSTARCH=arm64 \

      ./make.bash

  i386

  Building Go cmd/dist using /Users/jpap/Development/go/src/github.com/golang/go. (go1.16beta1 darwin/amd64)

  Building Go toolchain1 using /Users/jpap/Development/go/src/github.com/golang/go.

  Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.

  Building Go toolchain2 using go_bootstrap and Go toolchain1.

  Building Go toolchain3 using go_bootstrap and Go toolchain2.

  Building packages and commands for darwin/arm64.

  ---

  Installed Go for darwin/arm64 in /Users/jpap/go

  Installed commands in /Users/jpap/go/bin

  $ file ../bin/go

  ../bin/go: Mach-O 64-bit executable arm64

  $

This builds the darwin/amd64 go toolchain without error because GOHOSTARCH=arm64 overrides `uname -m` == i386 and CGO_* forces clang to target arm64 when it would otherwise target x86_64.

If I were to run the same example without CGO_*, the build fails with the intel syntax problem because clang targets x86_64.

  $ GOHOSTARCH=arm64 ./make.bash 

  i386

  Building Go cmd/dist using /Users/jpap/Development/go/src/github.com/golang/go. (go1.16beta1 darwin/amd64)

  Building Go toolchain1 using /Users/jpap/Development/go/src/github.com/golang/go.

  Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.

  Building Go toolchain2 using go_bootstrap and Go toolchain1.

  Building Go toolchain3 using go_bootstrap and Go toolchain2.

  Building packages and commands for darwin/arm64.

  # runtime/cgo

  gcc_arm64.S:28:16: error: brackets expression not supported on this target

  stp x29, x30, [sp, #-96]!

                ^

  [other errors snipped...]

Using an arm64 bootstrap compiler still gives you same failure, because clang still ends up running under x86_64.

  $ GOROOT_BOOTSTRAP=$HOME/go-bootstrap-arm64 GOHOSTARCH=arm64 ./make.bash 

  i386

  Building Go cmd/dist using /Users/jpap/go-bootstrap-arm64. (go1.16beta1 darwin/arm64)

  Building Go toolchain1 using /Users/jpap/go-bootstrap-arm64.

  Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.

  Building Go toolchain2 using go_bootstrap and Go toolchain1.

  Building Go toolchain3 using go_bootstrap and Go toolchain2.

  Building packages and commands for darwin/arm64.

  # runtime/cgo

  gcc_arm64.S:28:16: error: brackets expression not supported on this target

  stp x29, x30, [sp, #-96]!

                ^

  [other errors snipped...]

This is what I've called the x86_64 taint.  My apologies if my last message was unclear.

However, it is unclear to me how the system decides which architecture to run for the C compiler (or a universal binary in general). Apparently, the parent process' architecture seems to matter. The parent's parent (and more ancestors) also?

The Xcode toolchain ships with universal binaries, so each slice might have its own built-in defaults...

$ file `xcrun -f clang`

/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit executable x86_64] [arm64]

/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang (for architecture x86_64): Mach-O 64-bit executable x86_64

/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang (for architecture arm64): Mach-O 64-bit executable arm64

$ 

Is the -isysroot necessary? I don't think it is necessary if you are building on an M1 (even as a different architecture).

You are right: it is unnecessary when the SDK supports arch arm64.

At the time my build didn't work unless I had included the -isysroot flag.  Sadly, I can't reproduce it anymore: at the time I was struggling to get xocde-select working with Xcode vs. the Command Line Tools: it wouldn't switch to Xcode properly unless I rm'ed /Library/Developer/CommandLineTools so perhaps I ended up using both at one stage?

 

Maybe it would be helpful to ensure that the build is performed on an ARM64 environment and prevent AMD64 from "slipping in". This includes

Yes! This is the "x86_64 taint"... but it was hard to see in my case, because I was using the macOS default arm64 zsh and little did I know bash was installed as a x86_64 thin binary.  It becomes super clear if you add the arch command (or `uname -m`) into the make.bash script... perhaps that is a good motivation for a new CL?

 

- using an ARM64 bootstrap toolchain (otherwise generate one using bootstrap.bash)

I've also recommended the arm64 bootstrap, but even in that case, the x86_64 taint can still sting unless you use the environment variables given in the examples above.

 

- using an ARM64 bash. Maybe run make.bash with "arch -arm64 /bin/bash ./make.bash" ?

You need to be mindful of the interpreter directive in make.bash, which stung me due to an x86_64 bash installed via Homebrew which was further forward in my $PATH.  (As the examples above show.)  I uninstalled the Homebrew bash, and it solved my problem...

  $ echo $SHELL

  /bin/zsh

  $ arch

  arm64

  $ head -1 make.bash 

  #!/usr/bin/env bash

  $ /usr/bin/env bash -c arch

  i386

  $ /usr/local/bin/bash -c arch

  i386

  $ /bin/bash -c arch

  arm64

  $ brew uninstall bash

  Uninstalling /usr/local/Cellar/bash/5.1.4... (157 files, 10.9MB)

  Warning: The following may be bash configuration files and have not been removed!

  If desired, remove them manually with `rm -rf`:

    /usr/local/etc/bash_completion

    /usr/local/etc/bash_completion.d

  $ /usr/bin/env bash -c arch                                              

  arm64 # Succes!

jpap

[cherry]

 

- using an ARM64 bash. Maybe run make.bash with "arch -arm64 /bin/bash ./make.bash" ?

You need to be mindful of the interpreter directive in make.bash

Yeah, that's why I wrote absolute path /bin/bash. In the unlikely case that one replaces /bin/bash with an x86 bash, "arch -arm64 /bin/bash" will fail out loud.