Enable importing vendor packages for bootstrap tools like cmd/compile, cmd/asm, ...

[Qingwei Li]

Currently, the bootstrap process of `./make.bash` generate a project in /pkg/bootstrap/src/bootstrap and then install bootstrap tools like compile, asm, link, etc. However, it does not support vendor package now, i.e. the go.mod in bootstrap project contains no vendor packages. It's just plain as follows:

module bootstrap

go 1.22

If the cmd developers import vendor packages in cmd/compile, cmd/asm and other bootstrap tools, then it will fail because vendor packages are not written in go.mod.

So I try to solve the problem. In the last 2 commits of https://github.com/Lslightly/go/commits/bootstrap_vendor_copy/, I copy cmd/vendor to bootstrap/vendor and copy cmd/go.mod to bootstrap/go.mod with its module name and go version modified according to the minimal required bootstrap go version. I also change the `go install -tags="..." bootstrap/cmd/...` to `go install -tags="..." -mod=vendor bootstrap/cmd/...` to import packages in bootstrap/vendor directory.

After the above changes, it now supports importing vendor packages in cmd/compile by following steps:

1. import vendor packages in code, like src/cmd/compile/internal/x.go

2. run go get <pkg> and go mod vendor in src/cmd/ directory

3. run ./make.bash in src/ directory to rebuild the whole toolchain.

What's your idea about these two commits? Is the feature wanted, since it enables importing vendor packages in cmd/compile, cmd/asm?

Related issues: https://github.com/golang/go/issues/74563 and https://forum.golangbridge.org/t/go-cmd-compile-add-package/39214

[Ian Lance Taylor]

But cmd/compile/internal/gc imports cmd/internal/telemetry/counter
imports cmd/vendor/golang.org/x/telemetry/counter. So it at least
somewhat already works to have cmd/compile import vendored packages.

Ian

[Qingwei Li]

I notice that in cmd/internal/telemetry/counter/, there are counter.go and counter_bootstrap.go. They are different in the build tags. For counter.go, the build tag is "//go:build !cmd_go_bootstrap && !compiler_bootstrap". For counter_bootstrap.go, it is "//go:build cmd_go_bootstrap || compiler_bootstrap". counter_bootstrap.go does not import golang.org/x/telemetry/counter while counter.go does.

After generate project in pkg/bootstrap/src/bootstrap directory, the command `GOROOT_BOOTSTRAP/bin/go install -tags="math_big_pure_go compiler_bootstrap purego" bootstrap/cmd/...` is executed. So to build toolchain1, actually counter_bootstrap.go is used and no vendor packages are imported.

Only when building toolchain2 and toolchain3, counter.go and vendor package are used.

I conclude the above conclusion from the following log messages. I add log message in init function of cmd/internal/telemetry/counter/counter.go, counter_bootstrap.go and vendor/.../counter.go. When building toolchain1, counter_bootstrap.go is used. Therefore, when building bootstrap cmd/go and toolchain2, "counter_bootstrap` is printed. When building toolchain2 and toolchain3, "counter.go" and "vendor package" are used.

So instead of modifying "bootstrapBuildTools" function, another possible solution is to put features that import vendor packages in few go files and then tag them with "//go:build !cmd_go_bootstrap && !compiler_bootstrap" while providing _bootstrap.go version when bootstrapping?

Thanks for your pointer!

[Ian Lance Taylor]

On Sun, Jul 13, 2025 at 9:06 AM Qingwei Li <qingwe...@gmail.com> wrote:

I notice that in cmd/internal/telemetry/counter/, there are counter.go and counter_bootstrap.go. They are different in the build tags. For counter.go, the build tag is "//go:build !cmd_go_bootstrap && !compiler_bootstrap". For counter_bootstrap.go, it is "//go:build cmd_go_bootstrap || compiler_bootstrap". counter_bootstrap.go does not import golang.org/x/telemetry/counter while counter.go does.

After generate project in pkg/bootstrap/src/bootstrap directory, the command `GOROOT_BOOTSTRAP/bin/go install -tags="math_big_pure_go compiler_bootstrap purego" bootstrap/cmd/...` is executed. So to build toolchain1, actually counter_bootstrap.go is used and no vendor packages are imported.

Thanks, I missed that.

Since the bootstrap process works, and vendored packages are used in stages 2 and 3, what is the advantage of supporting vendored packages in stage 1?

Ian

[Qingwei Li]

Sorry for clicking the "reply to author" button instead of "reply to all" button.

So the final solution is to use build tags, "//go:build !cmd_go_bootstrap && !compiler_bootstrap" or "//go:build cmd_go_bootstrap || compiler_bootstrap".

Thanks for your replies.

---

On Tue, Jul 15, 2025 at 6:57 PM Qingwei Li wrote:
>
> The advantage might be consistency among stage1, 2 and 3? One code for 3 stages. Besides, developers don't need to worry about different behaviours among stage1 and 2/3.
>
> But I don't know the reason why bootstrap does not support vendored package. Since stage2 and 3 import vendored packages, security(e.g. supply chain attack) and maintainance(e.g. vendor package may require different version of go) issues exist no matter stage1 supports vendored packages, while using go:build tags in stage1 can add more complexity. (But the advantage of tags is that dependencies to vendored packages can be placed in a separated file, like counter_bootstrap.go/counter.go, which leads a cleaner project with all adapters placed in separate files. But it brings complexity as forementioned)
> I guess that if vendor package is used, then go1.4(the old bootstrap) can not build some vendor packages which requires higher version of go. But in recent versions of go, the version of GOROOT_BOOTSTRAP is higher and https://github.com/golang/go/issues/54265 says that bootstrap version will be bumped forward once a year, so the forementioned restriction can be relaxed now. I'm just guessing.

In general please write to the mailing list, not just to me. Thanks.

I don't see a reason to do any work to support vendored packages in
stage 1 unless there is some specific vendored package that make a big
difference. The capabilities of stage 1 are not very important. It's
stage 3 that matters. Stage 1 just has to be good enough to build
stage 2.

Ian