[pkgsite] internal/postgres: clean generated buf.build modules
4 views
Skip to first unread message
Jonathan Amsterdam (Gerrit)
Sep 9, 2026, 10:59:31 AMSep 9
to Hyang-Ah Hana Kim, Ethan Lee, goph...@pubsubhelper.golang.org, golang-co...@googlegroups.com
Attention needed from Ethan Lee and Hyang-Ah Hana Kim
Jonathan Amsterdam has uploaded the change for review![]( "Open in Gerrit")
Jonathan Amsterdam would like Hyang-Ah Hana Kim and Ethan Lee to review this change.
Commit message
internal/postgres: clean generated buf.build modules
module paths beginning with "buf.build/gen/go" seem to account for
about 4% of new modules, but they appear in the proxy as a side effect,
apparently, and nobody particularly cares about their documentation as
far as we know. So remove them from the DB.
We do it as part of cleaning instead of a one-off bulk delete so that
it's permanent, in case more such modules get inserted, and so that we
have an easy-to-fid artifact.
Change-Id: Ide768f12bcb0b0ae21b66549d5672a0c412113cf
Change diff
diff --git a/internal/postgres/clean.go b/internal/postgres/clean.go
index 9968747..3cf51ad 100644
--- a/internal/postgres/clean.go
+++ b/internal/postgres/clean.go
@@ -14,6 +14,8 @@
"golang.org/x/pkgsite/internal/derrors"
)
+const bufBuildPrefix = "buf.build/gen/go"
+
// GetModuleVersionsToClean returns module versions that can be removed from the database.
// Only module versions that were updated more than daysOld days ago will be considered.
// At most limit module versions will be returned.
@@ -47,11 +49,13 @@
WHERE requested_version IN ('master', 'main', 'dev.fuzz')
) vm_filtered ON m.module_path = vm_filtered.module_path AND m.version = vm_filtered.resolved_version
WHERE
+ (
m.version_type = 'pseudo'
AND CURRENT_TIMESTAMP - m.updated_at > make_interval(days => $1)
AND latest.path IS NULL
AND sd.module_path IS NULL
AND vm_filtered.module_path IS NULL
+ ) OR m.module_path LIKE '` + bufBuildPrefix + `%'
LIMIT $2
`
diff --git a/internal/postgres/clean_test.go b/internal/postgres/clean_test.go
index eeab9ea..a781210 100644
--- a/internal/postgres/clean_test.go
+++ b/internal/postgres/clean_test.go
@@ -26,12 +26,14 @@
want := []string{
"a...@v0.0.0-20190101000000-abcdef012345",
+ bufBuildPrefix + "a...@v1.0.0",
}
for _, mv := range append(want,
// These should not be cleaned.
"a...@v1.0.0", // tagged
"b...@v0.0.0-20190101000000-abcdef012345", // latest version
"b...@v0.0.0-20180101000000-abcdef012345", // 'main' in version_map (see UpsertVersionMap below)
+ "buf.build/gen/a...@v1.0.0", // non-gen buf.build module
) {
mod, ver, pkg := parseModuleVersionPackage(mv)
m := sample.Module(mod, ver, pkg)
Change information
Files:
- M internal/postgres/clean.go
- M internal/postgres/clean_test.go
Change size: XS
Delta: 2 files changed, 6 insertions(+), 0 deletions(-)
Related details
Attention is currently required from:
- Ethan Lee
- Hyang-Ah Hana Kim
Submit Requirements:
Code-Review
No-Unresolved-Comments
Review-Enforcement
TryBots-Pass
kokoro-CI-Passes
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Jonathan Amsterdam (Gerrit)
Sep 9, 2026, 11:08:11 AMSep 9
to goph...@pubsubhelper.golang.org, golang-co...@googlegroups.com
Attention needed from Ethan Lee, Hyang-Ah Hana Kim and Jonathan Amsterdam
Jonathan Amsterdam uploaded new patchset![]( "Open in Gerrit")
Attention is currently required from:
- Ethan Lee
- Hyang-Ah Hana Kim
- Jonathan Amsterdam
Submit Requirements:
Code-Review
No-Unresolved-Comments
Review-Enforcement
TryBots-Pass
kokoro-CI-Passes
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
kokoro (Gerrit)
Sep 9, 2026, 11:29:53 AMSep 9
to Jonathan Amsterdam, goph...@pubsubhelper.golang.org, golang...@luci-project-accounts.iam.gserviceaccount.com, Ethan Lee, Hyang-Ah Hana Kim, golang-co...@googlegroups.com
Attention needed from Ethan Lee, Hyang-Ah Hana Kim and Jonathan Amsterdam
kokoro voted kokoro-CI+1![]( "Open in Gerrit")
Kokoro presubmit build finished with status: SUCCESS
Logs at: https://source.cloud.google.com/results/invocations/9b55f24a-da4a-4e32-91c9-69ddf1328fa9
| kokoro-CI | +1 |
Related details
Attention is currently required from:
- Ethan Lee
- Hyang-Ah Hana Kim
- Jonathan Amsterdam
Submit Requirements:
Code-Review
No-Unresolved-Comments
Review-Enforcement
TryBots-Pass
kokoro-CI-Passes
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Ethan Lee (Gerrit)
Sep 9, 2026, 1:24:04 PMSep 9
to Jonathan Amsterdam, goph...@pubsubhelper.golang.org, kokoro, golang...@luci-project-accounts.iam.gserviceaccount.com, Hyang-Ah Hana Kim, golang-co...@googlegroups.com
Attention needed from Hyang-Ah Hana Kim and Jonathan Amsterdam
Ethan Lee added 3 comments![]( "Open in Gerrit")
File internal/postgres/clean.go
Line 17, Patchset 2 (Latest):
const bufBuildPrefix = "buf.build/gen/go"Ethan Lee . unresolved
This should have a short comment explaining why we have this prefix.
File internal/postgres/clean_test.go
Line 29, Patchset 2 (Latest):
bufBuildPrefix + "a...@v1.0.0",Ethan Lee . unresolved
Consider using the literal string here to ensure that the test independently verifies the value used by production code.
Line 29, Patchset 2 (Latest):
bufBuildPrefix + "a...@v1.0.0",Ethan Lee . unresolved
This results in the string 'buf.build/gen/g...@v1.0.0'. Include a slash: bufBuildPrefix + '/a...@v1.0.0'?
Related details
Attention is currently required from:
- Hyang-Ah Hana Kim
- Jonathan Amsterdam
Submit Requirements:
Code-Review
No-Unresolved-Comments
Review-Enforcement
TryBots-Pass
kokoro-CI-Passes
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Jonathan Amsterdam (Gerrit)
Sep 9, 2026, 1:56:18 PMSep 9
to goph...@pubsubhelper.golang.org, golang-co...@googlegroups.com
Attention needed from Ethan Lee and Hyang-Ah Hana Kim
Jonathan Amsterdam uploaded new patchset![]( "Open in Gerrit")
Jonathan Amsterdam uploaded patch set #3 to this change.
Following approvals got outdated and were removed:
- TryBots-Pass: LUCI-TryBot-Result+1 by golang...@luci-project-accounts.iam.gserviceaccount.com
Related details
Attention is currently required from:
- Ethan Lee
- Hyang-Ah Hana Kim
Submit Requirements:
Code-Review
No-Unresolved-Comments
Review-Enforcement
TryBots-Pass
kokoro-CI-Passes
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Jonathan Amsterdam (Gerrit)
Sep 9, 2026, 1:56:19 PMSep 9
to goph...@pubsubhelper.golang.org, kokoro, golang...@luci-project-accounts.iam.gserviceaccount.com, Ethan Lee, Hyang-Ah Hana Kim, golang-co...@googlegroups.com
Attention needed from Ethan Lee and Hyang-Ah Hana Kim
Jonathan Amsterdam added 3 comments![]( "Open in Gerrit")
File internal/postgres/clean.go
This should have a short comment explaining why we have this prefix.
Jonathan Amsterdam
Done
File internal/postgres/clean_test.go
Consider using the literal string here to ensure that the test independently verifies the value used by production code.
Jonathan Amsterdam
Done
This results in the string 'buf.build/gen/g...@v1.0.0'. Include a slash: bufBuildPrefix + '/a...@v1.0.0'?
Jonathan Amsterdam
Done
Related details
Attention is currently required from:
- Ethan Lee
- Hyang-Ah Hana Kim
Submit Requirements:
Code-Review
No-Unresolved-Comments
Review-Enforcement
TryBots-Pass
kokoro-CI-Passes
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Jonathan Amsterdam (Gerrit)
Sep 9, 2026, 1:57:51 PMSep 9
to goph...@pubsubhelper.golang.org, kokoro, golang...@luci-project-accounts.iam.gserviceaccount.com, Ethan Lee, Hyang-Ah Hana Kim, golang-co...@googlegroups.com
Attention needed from Ethan Lee and Hyang-Ah Hana Kim
Jonathan Amsterdam added 1 comment![]( "Open in Gerrit")
Patchset-level comments
Ethan Lee (Gerrit)
Sep 9, 2026, 2:10:11 PMSep 9
to Jonathan Amsterdam, goph...@pubsubhelper.golang.org, golang...@luci-project-accounts.iam.gserviceaccount.com, kokoro, Hyang-Ah Hana Kim, golang-co...@googlegroups.com
Attention needed from Hyang-Ah Hana Kim and Jonathan Amsterdam
Ethan Lee voted Code-Review+2![]( "Open in Gerrit")
Attention is currently required from:
- Hyang-Ah Hana Kim
- Jonathan Amsterdam
Submit Requirements:
Code-Review
No-Unresolved-Comments
Review-Enforcement
TryBots-Pass
kokoro-CI-Passes
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
kokoro (Gerrit)
Sep 9, 2026, 2:36:21 PMSep 9
to Jonathan Amsterdam, goph...@pubsubhelper.golang.org, Ethan Lee, golang...@luci-project-accounts.iam.gserviceaccount.com, Hyang-Ah Hana Kim, golang-co...@googlegroups.com
Attention needed from Hyang-Ah Hana Kim and Jonathan Amsterdam
kokoro voted kokoro-CI+1![]( "Open in Gerrit")
Kokoro presubmit build finished with status: SUCCESS
Related details
Attention is currently required from:
- Hyang-Ah Hana Kim
- Jonathan Amsterdam
Submit Requirements:
Code-Review
No-Unresolved-Comments
Review-Enforcement
TryBots-Pass
kokoro-CI-Passes
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Jonathan Amsterdam (Gerrit)
Sep 9, 2026, 2:49:39 PMSep 9
to goph...@pubsubhelper.golang.org, golang-...@googlegroups.com, kokoro, Ethan Lee, golang...@luci-project-accounts.iam.gserviceaccount.com, Hyang-Ah Hana Kim, golang-co...@googlegroups.com
Jonathan Amsterdam submitted the change![]( "Open in Gerrit")
Change information
Commit message:
internal/postgres: clean generated buf.build modules
module paths beginning with "buf.build/gen/go" seem to account for
about 4% of new modules, but they appear in the proxy as a side effect,
apparently, and nobody particularly cares about their documentation as
far as we know. So remove them from the DB.
We do it as part of cleaning instead of a one-off bulk delete so that
it's permanent, in case more such modules get inserted, and so that we
have an easy-to-find artifact.
We don't try to clean all excluded modules because what if someday
somebody wants to unexclude something? That could result in a lot of
work. But maybe that's not a good argument. Maybe we should be cleaning
them.
Change-Id: Ide768f12bcb0b0ae21b66549d5672a0c412113cf
Reviewed-by: Ethan Lee <etha...@google.com>
LUCI-TryBot-Result: golang...@luci-project-accounts.iam.gserviceaccount.com <golang...@luci-project-accounts.iam.gserviceaccount.com>
kokoro-CI: kokoro <noreply...@google.com>
Files:
- M internal/postgres/clean.go
- M internal/postgres/clean_test.go
Change size: XS
Delta: 2 files changed, 9 insertions(+), 0 deletions(-)
Branch: refs/heads/master
Submit Requirements:
Code-Review: +2 by Ethan Lee
TryBots-Pass: LUCI-TryBot-Result+1 by golang...@luci-project-accounts.iam.gserviceaccount.com
kokoro-CI-Passes: kokoro-CI+1 by kokoro
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Jonathan Amsterdam (Gerrit)
6:11 PM (4 hours ago) 6:11 PM
to goph...@pubsubhelper.golang.org, kokoro, Ethan Lee, golang...@luci-project-accounts.iam.gserviceaccount.com, Hyang-Ah Hana Kim, golang-co...@googlegroups.com
Jonathan Amsterdam has created a revert of this change![]( "Open in Gerrit")
Related details
Attention set is empty
Submit Requirements:
Code-Review
No-Unresolved-Comments
Review-Enforcement
TryBots-Pass
kokoro-CI-Passes
| Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. |
Reply all
Reply to author
Forward
0 new messages