[build] all: migrate to 'gcloud storage' from gsutil
2 views
Skip to first unread message
Dmitri Shuralyov (Gerrit)
Feb 25, 2026, 11:22:40 AM (2 days ago) Feb 25
to Carlos Amedee, goph...@pubsubhelper.golang.org, Dmitri Shuralyov, golang-co...@googlegroups.com
Attention needed from Carlos Amedee
Dmitri Shuralyov has uploaded the change for review![]( "Open in Gerrit")
Dmitri Shuralyov would like Carlos Amedee to review this change.
Commit message
all: migrate to 'gcloud storage' from gsutil
At https://docs.cloud.google.com/storage/docs/gsutil#should-you-use it
is suggested to prefer using 'gcloud storage' instead of gsutil. Do so.
This was originally brought to my attention via the description of the
pull request golang/build#116, and it replaces that change.
Change-Id: I5273aa61155e61a1645d45ee679222f0a9c959b3
Change diff
diff --git a/cmd/upload/upload.go b/cmd/upload/upload.go
index 38745a5..24cf68a 100644
--- a/cmd/upload/upload.go
+++ b/cmd/upload/upload.go
@@ -4,7 +4,7 @@
// The upload command writes a file to Google Cloud Storage. It's used
// exclusively by the Makefiles in the Go project repos. Think of it
-// as a very light version of gsutil or gcloud, but with some
+// as a very light version of 'gcloud storage', but with some
// Go-specific configuration knowledge baked in.
package main
diff --git a/env/plan9/README b/env/plan9/README
index a4fe028..1e70a47 100644
--- a/env/plan9/README
+++ b/env/plan9/README
@@ -24,7 +24,7 @@
$ ./make.bash
Then:
- $ gsutil cp -a public-read plan9-386-gce.tar.gz gs://go-builder-data/plan9-386-gce.tar.gz
+ $ gcloud storage cp -a public-read plan9-386-gce.tar.gz gs://go-builder-data/plan9-386-gce.tar.gz
Then:
$ gcloud compute --project symbolic-datum-552 images create plan9-386-v6 --source-uri gs://go-builder-data/plan9-386-gce.tar.gz
diff --git a/env/windows-arm64/azure/setupAndRunAllDotBat.sh b/env/windows-arm64/azure/setupAndRunAllDotBat.sh
index 45a6f34..6f7a7b7 100644
--- a/env/windows-arm64/azure/setupAndRunAllDotBat.sh
+++ b/env/windows-arm64/azure/setupAndRunAllDotBat.sh
@@ -14,7 +14,7 @@
# invoker to enter the VM account password several times. TODO: use -M and
# -S ssh flags to avoid reauthentication.
#
-# This script also uses "gsutil" to copy things from the go-builder GCS bucket
+# This script also uses "gcloud storage" to copy things from the go-builder GCS bucket
# as part of the setup.
#
#-----------------------------
@@ -51,8 +51,8 @@
function copy_from_go_builder_data() {
local FILE="$1"
local TGT="$2"
- echo "... executing: gsutil cp gs://go-builder-data/${FILE} $TGT"
- gsutil cp gs://go-builder-data/${FILE} $TGT
+ echo "... executing: gcloud storage cp gs://go-builder-data/${FILE} $TGT"
+ gcloud storage cp gs://go-builder-data/${FILE} $TGT
if [ $? != 0 ]; then
echo "error: copy from gs://go-builder-data/${FILE} failed, aborting"
exit 1
diff --git a/internal/relui/workflows.go b/internal/relui/workflows.go
index aacc58e..3b97290 100644
--- a/internal/relui/workflows.go
+++ b/internal/relui/workflows.go
@@ -979,7 +979,7 @@
}
script := fmt.Sprintf(`
-gsutil cp %q source.tgz
+gcloud storage cp %q source.tgz
mkdir go
tar -xf source.tgz -C go
echo -ne %q > go/VERSION
@@ -1105,7 +1105,7 @@
makeEnv = append(makeEnv, "GOOS="+target.GOOS, "GOARCH="+target.GOARCH)
script := fmt.Sprintf(`
-gsutil cp %q src.tar.gz
+gcloud storage cp %q src.tar.gz
tar -xf src.tar.gz
(cd go/src && %v ./make.bash -distpack)
(cd go/pkg/distpack && tar -czf ../../../distpacks.tar.gz *)
@@ -1139,7 +1139,7 @@
// for testing. In particular, Windows doesn't seem to like ./foo.exe,
// so we have to run it unadorned with . on PATH.
script := fmt.Sprintf(
- `gsutil cat %s | tar -xzf - && cd go/src && make.bat -distpack && cd ../pkg/distpack && tar -czf - * | gsutil cp - %s`,
+ `gcloud storage cat %s | tar -xzf - && cd go/src && make.bat -distpack && cd ../pkg/distpack && tar -czf - * | gcloud storage cp - %s`,
b.ScratchFS.URL(ctx, source.Scratch), b.ScratchFS.URL(ctx, scratchFile))
env := map[string]string{
diff --git a/internal/task/cloudbuild.go b/internal/task/cloudbuild.go
index 0570ff7..5379bc1 100644
--- a/internal/task/cloudbuild.go
+++ b/internal/task/cloudbuild.go
@@ -43,14 +43,14 @@
// RunScript runs the given script under bash -eux -o pipefail in
// ScriptProject. Outputs are collected into the build's ResultURL,
// readable with ResultFS. The script will have the latest stable version of Go
- // and some version of gsutil on $PATH.
+ // and some version of gcloud on $PATH.
// If gerritProject is provided, the script operates within a checkout of the
// latest commit on the default branch of that repository.
RunScript(ctx context.Context, script string, gerritProject string, outputs []string) (CloudBuild, error)
// RunCustomSteps is a low-level API that provides direct control over
// individual Cloud Build steps. It creates a random result directory
// and provides that as a parameter to the steps function, so that it
- // may write output to it with 'gsutil cp' for accessing via ResultFS.
+ // may write output to it with 'gcloud storage cp' for accessing via ResultFS.
// Prefer RunScript for simpler scenarios.
// Reference: https://cloud.google.com/build/docs/build-config-file-schema
RunCustomSteps(ctx context.Context, steps func(resultURL string) []*cloudbuildpb.BuildStep, opts *CloudBuildOptions) (CloudBuild, error)
@@ -125,7 +125,7 @@
var saveOutputsScript strings.Builder
saveOutputsScript.WriteString(cloudBuildClientScriptPrefix)
for _, out := range outputs {
- fmt.Fprintf(&saveOutputsScript, "gsutil cp %q %q\n", out, resultURL+"/"+strings.TrimPrefix(out, "./"))
+ fmt.Fprintf(&saveOutputsScript, "gcloud storage cp %q %q\n", out, resultURL+"/"+strings.TrimPrefix(out, "./"))
}
var steps []*cloudbuildpb.BuildStep
@@ -143,12 +143,12 @@
Script: cloudBuildClientDownloadGoScript,
},
&cloudbuildpb.BuildStep{
- Name: "gcr.io/cloud-builders/gsutil",
+ Name: "gcr.io/cloud-builders/gcloud",
Script: cloudBuildClientScriptPrefix + script,
Dir: dir,
},
&cloudbuildpb.BuildStep{
- Name: "gcr.io/cloud-builders/gsutil",
+ Name: "gcr.io/cloud-builders/gcloud",
Script: saveOutputsScript.String(),
Dir: dir,
},
diff --git a/internal/task/fakes.go b/internal/task/fakes.go
index 7c03165..2186a19 100644
--- a/internal/task/fakes.go
+++ b/internal/task/fakes.go
@@ -918,13 +918,13 @@
var _ CloudBuildClient = (*FakeCloudBuild)(nil)
-const fakeGsutil = `
+const fakeGcloud = `
#!/bin/bash -eux
-case "$1" in
-"cp")
- in=$2
- out=$3
+case "$1 $2" in
+"storage cp")
+ in=$3
+ out=$4
if [[ $in == '-' ]]; then
in=/dev/stdin
fi
@@ -935,8 +935,8 @@
mkdir -p "${dir#file://}"
cp "${in#file://}" "${out#file://}"
;;
-"cat")
- cat "${2#file://}"
+"storage cat")
+ cat "${3#file://}"
;;
*)
echo unexpected command $@ >&2
@@ -960,7 +960,7 @@
func NewFakeCloudBuild(t *testing.T, gerrit *FakeGerrit, project string, allowedTriggers map[string]map[string]string, fakeBinaries ...FakeBinary) *FakeCloudBuild {
toolDir := t.TempDir()
- if err := os.WriteFile(filepath.Join(toolDir, "gsutil"), []byte(fakeGsutil), 0777); err != nil {
+ if err := os.WriteFile(filepath.Join(toolDir, "gcloud"), []byte(fakeGcloud), 0777); err != nil {
t.Fatal(err)
}
@@ -1213,7 +1213,7 @@
if err := os.WriteFile(filepath.Join(toolDir, "go"), []byte(fakeGo), 0777); err != nil {
t.Fatal(err)
}
- if err := os.WriteFile(filepath.Join(toolDir, "gsutil"), []byte(fakeGsutil), 0777); err != nil {
+ if err := os.WriteFile(filepath.Join(toolDir, "gcloud"), []byte(fakeGcloud), 0777); err != nil {
t.Fatal(err)
}
return &FakeSwarmingClient{
diff --git a/internal/task/releasegopls.go b/internal/task/releasegopls.go
index 6eedc09..6533912 100644
--- a/internal/task/releasegopls.go
+++ b/internal/task/releasegopls.go
@@ -756,8 +756,8 @@
go run -C extension tools/generate.go -w -tools -gopls
`)
for _, file := range files {
- fmt.Fprintf(&updateScript, "gsutil cp %s %s/%s\n", file, resultURL, file)
- fmt.Fprintf(&updateScript, "gsutil cp %s.before %s/%s.before\n", file, resultURL, file)
+ fmt.Fprintf(&updateScript, "gcloud storage cp %s %s/%s\n", file, resultURL, file)
+ fmt.Fprintf(&updateScript, "gcloud storage cp %s.before %s/%s.before\n", file, resultURL, file)
}
return []*cloudbuildpb.BuildStep{
{
@@ -782,7 +782,7 @@
Dir: "vscode-go",
},
{
- Name: "gcr.io/cloud-builders/gsutil",
+ Name: "gcr.io/cloud-builders/gcloud",
Script: fmt.Sprintf(updateScript.String(), version),
Dir: "vscode-go",
},
diff --git a/internal/task/releasevscodego.go b/internal/task/releasevscodego.go
index 2507469..1383e55 100644
--- a/internal/task/releasevscodego.go
+++ b/internal/task/releasevscodego.go
@@ -198,8 +198,8 @@
Dir: "vscode-go",
},
{
- Name: "gcr.io/cloud-builders/gsutil",
- Args: []string{"cp", "output.log", fmt.Sprintf("%s/output.log", resultURL)},
+ Name: "gcr.io/cloud-builders/gcloud",
+ Args: []string{"storage", "cp", "output.log", fmt.Sprintf("%s/output.log", resultURL)},
Dir: "vscode-go",
},
}
@@ -328,7 +328,7 @@
var saveScript strings.Builder
saveScript.WriteString(cloudBuildClientScriptPrefix)
for _, file := range []string{"npm-output.log", "go-package-output.log", vsixFileName(release, prerelease)} {
- fmt.Fprintf(&saveScript, "gsutil cp %s %s/%s\n", file, resultURL, file)
+ fmt.Fprintf(&saveScript, "gcloud storage cp %s %s/%s\n", file, resultURL, file)
}
return []*cloudbuildpb.BuildStep{
{
@@ -350,7 +350,7 @@
Dir: "vscode-go/extension",
},
{
- Name: "gcr.io/cloud-builders/gsutil",
+ Name: "gcr.io/cloud-builders/gcloud",
Script: saveScript.String(),
Dir: "vscode-go/extension",
},
@@ -392,7 +392,7 @@
var saveScript strings.Builder
saveScript.WriteString(cloudBuildClientScriptPrefix)
for _, file := range []string{"npm-output.log", "go-publish-output.log", vsixFileName(release, "")} {
- fmt.Fprintf(&saveScript, "gsutil cp %s %s/%s\n", file, resultURL, file)
+ fmt.Fprintf(&saveScript, "gcloud storage cp %s %s/%s\n", file, resultURL, file)
}
return []*cloudbuildpb.BuildStep{
{
@@ -409,7 +409,7 @@
// files to publish.
// TODO(hxjiang): write the vsix file to a separate gcs bucket and read
// it from there.
- Name: "gcr.io/cloud-builders/gsutil",
+ Name: "gcr.io/cloud-builders/gcloud",
Args: []string{"cp", build.ResultURL + "/" + vsixFileName(release, ""), "."},
Dir: "vscode-go/extension",
},
@@ -420,7 +420,7 @@
SecretEnv: []string{"VSCE_PAT"},
},
{
- Name: "gcr.io/cloud-builders/gsutil",
+ Name: "gcr.io/cloud-builders/gcloud",
Script: saveScript.String(),
Dir: "vscode-go/extension",
},
@@ -882,7 +882,7 @@
"npx-output.log",
"npm-output.log",
} {
- fmt.Fprintf(&saveScriptFmt, "gsutil cp %[1]s %[2]s/%[1]s\n", file, resultURL)
+ fmt.Fprintf(&saveScriptFmt, "gcloud storage cp %[1]s %[2]s/%[1]s\n", file, resultURL)
}
return []*cloudbuildpb.BuildStep{
@@ -901,7 +901,7 @@
Dir: "vscode-go/extension",
},
{
- Name: "gcr.io/cloud-builders/gsutil",
+ Name: "gcr.io/cloud-builders/gcloud",
Script: saveScriptFmt.String(),
Dir: "vscode-go/extension",
},
diff --git a/internal/task/swarming.go b/internal/task/swarming.go
index eb224ca..f9281b9 100644
--- a/internal/task/swarming.go
+++ b/internal/task/swarming.go
@@ -15,7 +15,7 @@
type SwarmingClient interface {
// RunTask runs script on a machine running port with env set.
- // The script will have the latest version of Go and some version of gsutil
+ // The script will have the latest version of Go and some version of gcloud
// on $PATH. To facilitate Windows/Unix compatibility, . will be at the end
// of $PATH.
RunTask(ctx context.Context, dims map[string]string, script string, env map[string]string) (string, error)
Change information
Files:
- M cmd/upload/upload.go
- M env/plan9/README
- M env/windows-arm64/azure/setupAndRunAllDotBat.sh
- M internal/relui/workflows.go
- M internal/task/cloudbuild.go
- M internal/task/fakes.go
- M internal/task/releasegopls.go
- M internal/task/releasevscodego.go
- M internal/task/swarming.go
Change size: M
Delta: 9 files changed, 35 insertions(+), 35 deletions(-)
Related details
Attention is currently required from:
- Carlos Amedee
Submit Requirements:
Code-Review
No-Unresolved-Comments
Review-Enforcement
TryBots-Pass
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Dmitri Shuralyov (Gerrit)
Feb 25, 2026, 11:27:24 AM (2 days ago) Feb 25
to Dmitri Shuralyov, goph...@pubsubhelper.golang.org, Go LUCI, Carlos Amedee, golang-co...@googlegroups.com
Attention needed from Carlos Amedee
Dmitri Shuralyov voted Code-Review+1![]( "Open in Gerrit")
| Code-Review | +1 |
Related details
Attention is currently required from:
- Carlos Amedee
Submit Requirements:
Code-Review
No-Unresolved-Comments
Review-Enforcement
TryBots-Pass
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Carlos Amedee (Gerrit)
Feb 25, 2026, 3:23:55 PM (2 days ago) Feb 25
to Dmitri Shuralyov, goph...@pubsubhelper.golang.org, Go LUCI, Dmitri Shuralyov, golang-co...@googlegroups.com
Attention needed from Dmitri Shuralyov
Carlos Amedee voted and added 1 comment![]( "Open in Gerrit")
Votes added by Carlos Amedee
| Code-Review | +2 |
1 comment
Patchset-level comments
File-level comment, Patchset 1 (Latest):
Carlos Amedee . resolved
This looks correct to me. I'm guessing we won't see if any of these changes causes a failure until we run them. Which is fine.
Related details
Attention is currently required from:
- Dmitri Shuralyov
Submit Requirements:
Code-Review
No-Unresolved-Comments
Review-Enforcement
TryBots-Pass
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Dmitri Shuralyov (Gerrit)
4:09 PM (2 hours ago) 4:09 PM
to Dmitri Shuralyov, goph...@pubsubhelper.golang.org, Carlos Amedee, Go LUCI, Dmitri Shuralyov, golang-co...@googlegroups.com
Dmitri Shuralyov voted and added 1 comment![]( "Open in Gerrit")
Votes added by Dmitri Shuralyov
| Hold | +1 |
1 comment
Patchset-level comments
Carlos Amedee . resolved
This looks correct to me. I'm guessing we won't see if any of these changes causes a failure until we run them. Which is fine.
Dmitri Shuralyov
Agreed. Since this is a refactor and can probably wait a bit, putting it on hold until a better time to land it and follow up as needed. Thanks.
Related details
Attention set is empty
Submit Requirements:
Code-Review
No-Holds
No-Unresolved-Comments
Review-Enforcement
TryBots-Pass
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Reply all
Reply to author
Forward
0 new messages