Editing & Debugging the compiler code

[Mark Diener]

Hello Golang devs:

There is a multi-step process in /src/cmd/dist/dist.go that bootstraps building the compiler so that it can use updated version of the compiler to build the final artifact.  

But I would like to just set the compiler to build the source from the currently installed golang version and not execute 3 or 4 bootstrap passes of the compiler to produce the final artifact.

Any comments on how to bypass the multi-step compiler build process in /src/cmd/dist/dist.go and just have a single command to do a single pass build so that I can point my debugger at the final artifact.

Any responses appreciated.

Build process below:

marco@Marks-MacBook-Pro src % ./make.bash

Building Go cmd/dist using /opt/homebrew/Cellar/go/1.19.1/libexec. (go1.19.1 darwin/arm64)
Building Go toolchain1 using /opt/homebrew/Cellar/go/1.19.1/libexec.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.
Building packages and commands for darwin/arm64.
Installed Go for darwin/arm64 in /Users/sdev/golang/go-compiler
Installed commands in /Users/sdev/golang/go-compiler/bin

[Alessandro Arzilli]

cd $GOROOT/src/cmd/compile
go install

> --
> You received this message because you are subscribed to the Google Groups "golang-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-dev/ecc2a5bf-9f51-4582-981b-c0739f9fc2d4n%40googlegroups.com.

[Mark Diener]

Hello Alesandro:

What is the binary/executable produced by "go install" when you execute it from $GOROOT/src/cmd/compile?
(This seems to build only a small portion of the full go binary, the CmdInstall portion..)

Is there an alternative to make.bash that will build the full go binary but not use a multi-pass concept but maybe use the compiler built in the last stage of make.bash
to compile the final artifact so that I can then load it into the debugger?

Thanks for your previous response and look forward to having a dialogue

Best,  Mark

[David Chase]

It “just works” because of the Go builder’s dependency management, though you have to be careful about what you are testing or repeat the process.

So:

< tweak the compiler, which is for the Go command on your path >
go install cmd/compile # old compiler builds new compiler
go test something        # rebuilds something and everything it depends on
go install cmd/compile # new compiler rebuilds new compiler and everything it depends on (that wasn’t recompiled when you tested “something"

There are corner cases if you make code in the compiler depend on environment variables for testing etc that the builder doesn’t know about.

David

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-dev/6e946a3f-fe64-4516-84b7-deb185151371n%40googlegroups.com.

[Alessandro Arzilli]

The go toolchain isn't a single binary, there are multiple executables, you
can see them in $GOROOT/pkg/tool/$GOOS_$GOARCH/.

You are probably thinking of the 'go' command, that one is in src/cmd/go,
you can use 'go install' and 'go build' (and 'dlv debug') in that directory.
But it isn't a compiler, it's more like make (although it does a lot of
extra things), it will call cmd/compile and cmd/link as needed to compile
source code.

You can look at what it does with 'go build -x'.

Usually when I want to debug the compiler I get the arguments from 'go build -x'
and then run:
$ dlv debug -- <compiler arguments...>
in src/cmd/compile (I actually use gdlv but it doesn't matter).

As far as I am aware there isn't a version of make.bash that does fewer
step, I might be ignorant.

> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-dev/6e946a3f-fe64-4516-84b7-deb185151371n%40googlegroups.com.

[Mark Diener]

Hello Alessandro:

That certainly modified my perspective, understanding now that there are different binaries based on the "base.Command" array.

David Chase also gave me an answer and I am trying to process that while digesting your messages.
(Sorry David, I answered your message directly thru email, and not this group channel, I will copy it to this list right after this message)

You remember that moment in Star Wars where Obi Wan Kenobi handed Luke Skywalker a lightsaber for the first time and Luke flipped it on
but did not really appreciate what he held in his hands?  I am kind of like a young Luke, both David Chase and yourself are trying your best to describe
how the lightsaber works that you have presented me, patience young Jedi.

Best,

Mark

[Mark Diener]

Good morning David.

I am having a bit of a rub trying to discover how to make your suggestion actionable.

From my observations of the build process, the 2 artifacts produced by make.bash are put in the $GOROOT/bin: go and gofmt.

Deleting these 2 files and running the "go install /cmd/compile" command does not seem to indicate WHERE any artifacts produced are located, but they certainly are not regenerated in $GOROOT/bin.

When I execute "go install /cmd/compile" , where is the binary output of that command placed?

Do you have a suggestion of how I might expand the scope of this so that I can set a breakpoint inside of  src/cmd/go/main.go -> func main() so that I can step into the full go binary?

Once I have some control over the full binary, maybe I can optimize my developer experience by using your tricks to only work on a small portion of the code base at a time like using
the command "go test <something>" running unit tests on a small piece of the code.

My guestimate is that if I can intercept the 3rd stage compile of the make.bash script, I can use that version of the compiler to build the 4th stage of the source and then I can set my debugger
in visual studio code to launch the main.go -> func main() with a single pass compile.

My MacOS environment is:

which go <enter> => /opt/homebrew/bin/go
/opt/homebrew/bin/go version => go version go1.19.1 darwin/arm64

Best,

Mark Diener

On Saturday, October 1, 2022 at 8:18:07 AM UTC-6 alessandr...@gmail.com wrote:

[Ian Lance Taylor]

On Sat, Oct 1, 2022 at 7:43 AM Mark Diener <markdi...@gmail.com> wrote:
>
> When I execute "go install /cmd/compile" , where is the binary output of that command placed?

Typically in $GOROOT/pkg/tool/${GOOS}_${GOARCH}. You can see the
exact location on your system by running "go tool -n compile". You
can invoke the new binary via "go tool compile" (or you can just run
it directly).

Ian

[Mark Diener]

Hey David Chase, that last post was just so that I could make public my mistaken email send directly to your email instead of using the google groups area here.

Alessandro gave me a bit of a heads up that the go command actually has a bunch of tools in pkg/tool/darwin_arm64 like compile and cgo, which are distinct binaries separate
from the "go" binary located in $GOROOT/bin/go (and gofmt).

Lance Taylor reinforced Alessandro's message.

So really to make code edits to the compiler and then set breakpoints in the compiler binary, I need to be able to set up test cases to learn how the build pipeline works.

Any suggestions on any additional materials on the ABI format for golang?   This includes getting a feeling for the AST syntax tree and the other components that ultimately yield CPU architecture specific machine language, but for now, I am just using darwing/arm64 since I am building on an M1 Macos computer.

When the golang team at google onboards a new engineer, is there a set of materials so the young Jedi in training can ramp up their grasp of how the compiler works, etc?  Basically, consider me onboarding onto the team without expecting a paycheck or attending the daily standup.

Maybe I am not the only community member that wants to ramp up on the technical details of how the compiler/build, but find that just reading the ABI readme file located somewhere in the source tree to does not really make the material come alive.

Thanks to David Chase, Lance Tayler, and Alessandro.

Best,

Mark

[Ian Lance Taylor]

On Sat, Oct 1, 2022 at 9:03 AM Mark Diener <markdi...@gmail.com> wrote:
>
> When the golang team at google onboards a new engineer, is there a set of materials so the young Jedi in training can ramp up their grasp of how the compiler works, etc? Basically, consider me onboarding onto the team without expecting a paycheck or attending the daily standup.
>
> Maybe I am not the only community member that wants to ramp up on the technical details of how the compiler/build, but find that just reading the ABI readme file located somewhere in the source tree to does not really make the material come alive.

See cmd/compile/README.md.

Ian

[Mark Diener]

Hello Golang-devs:  (Ian Taylor!)

After cloning the git repo: git clone https://go.googlesource.com/go

There is obviously some additional layers of complexity in the build sequence

I am able to build master branch 1.20.x using:  /src/make.bash.

marco@MacBook-Pro darwin_arm64 % /Users/marco/sdev/go120/pkg/tool/compile -V
compile version devel go1.20-ae9341ede9 Sat Oct 29 23:22:01 2022 -0600

But when I change to /src/cmd/compile directory and issue: "go build" from inside that compiler directory, it creates a "compile" binary executable in that directory, but

it is not the code from the current branch and builds 1.19.2.

marco@MacBook-Pro compile % ./compile -V

compile version go1.19.2                                                                                                                          

Anybody have a suggestion on how to build the /src/cmd/compile code in isolation, and maybe from visual studio code

and set a breakpoint and debug the compiler code?

All responses welcome.

Mark

[Bryan C. Mills]

You are likely running into https://go.dev/issue/33138.

The solution is to ensure that when you run 'go build', you are (a) not setting GOROOT to the wrong value (generally best to leave it completely unset), and (b) using the 'go' command from $GOROOT/bin of the same GOROOT directory.

--

You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-dev/df4ba3e6-a72a-4244-9689-62ad9fc0fa75n%40googlegroups.com.

[Mark Diener]

Hello Bryan.

Thanks for your response.  I am not still clear from your comment how to control my GOROOT so that the correct code is included in the compilation.

I highlighted the problem in yellow & red in the code snippet below, I need it to read go120 instead of go119 (which is the make.bash toolchain)

marco@Marks-MacBook-Pro compile % go build -v -x
....output truncated.....

cd /Users/sdev/godev/go120/src/cmd/compile

/Users/sdev/godev/go119/pkg/tool/darwin_arm64/compile -o $WORK/b001/_pkg_.a -trimpath "$WORK/b001=>" -p main -std -complete -buildid 4X6kR3ykcIABSrWAWbtV/4X6kR3ykcIABSrWAWbtV -goversion go1.19.2 -shared -c=4 -nolocalimports -importcfg $WORK/b001/importcfg -pack /Users/sdev/godev/go119/src/cmd/compile/doc.go /Users/sdev/godev/go119/src/cmd/compile/main.go

There is no GOROOT or GOPATH defined in my linux/macos environment, just in the PATH to go119/bin/go command. "go env" produces the following:

GOROOT="/Users/sdev/godev/go119"

We hopefully can solve this with a source directory being built outside of the toolchain GOROOT path and actually compile the code in the directory within which the build command

is being executed. 

Mark Diener

[Bryan C. Mills]

When you are in go120/src, you cannot use go119/bin/go to build the standard library: from the perspective of go119/bin/go, go120/src is just another module that happens to be named "std".

Instead, when you are working within your go120 GOROOT, you must use go120/bin/go.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-dev/CAC3g_BGVyODi40mXNQVfoX02Gg5X9-tmfiBkvnN7Zm0B8pntzQ%40mail.gmail.com.

[Than McIntosh]

Hello,

The convention for developers is to arrange for the "bin" directory of the Go tree you're building  to be earlier in your path than the bootstrap version. That way once you've finished running make.bash, your GOROOT will switch to the dev version, which is what you want.

Appearing below is a small shell script with a demonstration, along with a transcript of what I see when I run the script on my machine.

HTH,

Than

---

#!/bin/bash

# this assumes that your bootstrap "go" is located in /tmp/go1.19
export PATH=/tmp/go1.19/bin:/usr/local/bin:/usr/bin
cd /tmp
set -xe
echo PATH is $PATH
which go
go version
go env GOROOT
rm -rf devgo
git clone https://go.googlesource.com/go devgo
export PATH=/tmp/devgo/bin:/tmp/go1.19/bin:/usr/local/bin:/usr/bin
echo revised PATH is now $PATH
which go
go env GOROOT
cd devgo/src
bash make.bash
echo make.bash complete
# inform shell that "go" is coming from a new place
hash -r
ls /tmp/devgo/bin/go
which go
go version
go env GOROOT
cd cmd/compile
rm -f compile.exe
go clean -cache
go build -x -v -o compile.exe . 1> out.txt 2>&1
ls compile.exe
# Expected output of the fgrep below:
# /tmp/devgo/pkg/tool/linux_amd64/compile ... -pack ./doc.go ./main.go
fgrep ./doc.go out.txt | fgrep -e "-p main"

Output when I run it:

+ cd /tmp
+ echo PATH is /tmp/go1.19/bin:/usr/local/bin:/usr/bin
PATH is /tmp/go1.19/bin:/usr/local/bin:/usr/bin
+ which go
/tmp/go1.19/bin/go
+ go version
go version go1.19.2 linux/amd64
+ go env GOROOT
/tmp/go1.19
+ rm -rf devgo
+ git clone https://go.googlesource.com/go devgo
Cloning into 'devgo'...
Updating files:  78% (9293/11903)

...
Updating files: 100% (11903/11903), done.
+ export PATH=/tmp/devgo/bin:/tmp/go1.19/bin:/usr/local/bin:/usr/bin
+ PATH=/tmp/devgo/bin:/tmp/go1.19/bin:/usr/local/bin:/usr/bin
+ echo revised PATH is now /tmp/devgo/bin:/tmp/go1.19/bin:/usr/local/bin:/usr/bin
revised PATH is now /tmp/devgo/bin:/tmp/go1.19/bin:/usr/local/bin:/usr/bin
+ which go
/tmp/go1.19/bin/go
+ go env GOROOT
/tmp/go1.19
+ cd devgo/src
+ bash make.bash
Building Go cmd/dist using /tmp/go1.19. (go1.19.2 linux/amd64)
Building Go toolchain1 using /tmp/go1.19.

Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.

Building packages and commands for linux/amd64.
---
Installed Go for linux/amd64 in /tmp/devgo
Installed commands in /tmp/devgo/bin
+ echo make.bash complete
make.bash complete
+ hash -r
+ ls /tmp/devgo/bin/go
/tmp/devgo/bin/go
+ which go
/tmp/devgo/bin/go
+ go version
go version devel go1.20-30b240b102 Wed Nov 2 13:14:08 2022 +0000 linux/amd64
+ go env GOROOT
/tmp/devgo
+ cd cmd/compile
+ rm -f compile.exe
+ go clean -cache
+ go build -x -v -o compile.exe .
+ ls compile.exe
compile.exe
+ fgrep compile/doc.go out.txt
/tmp/devgo/pkg/tool/linux_amd64/compile -o $WORK/b001/_pkg_.a -trimpath "$WORK/b001=>" -p main -std -complete -buildid 4k-ivyMCmTiWKlfCGRf1/4k-ivyMCmTiWKlfCGRf1 -c=4 -nolocalimports -importcfg $WORK/b001/importcfg -pack ./doc.go ./main.go

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-dev/CAC3g_BGVyODi40mXNQVfoX02Gg5X9-tmfiBkvnN7Zm0B8pntzQ%40mail.gmail.com.