[vulndb] all: Remove O(N^2) alias check and redundant disk I/O

5 views
Skip to first unread message

Ethan Lee (Gerrit)

unread,
Dec 26, 2025, 1:46:44 PM (2 days ago) Dec 26
to goph...@pubsubhelper.golang.org, golang-co...@googlegroups.com

Ethan Lee has uploaded the change for review

Commit message

all: Remove O(N^2) alias check and redundant disk I/O

TestLintReports was hitting the 10-minute timeout in CI due to scaling
bottlenecks as the vulnerability database grows. This change
optimizes the test to scale linearly with the number of reports.

- Eliminate redundant disk I/O by retrieving reports directly from
  the report.LocalClient's memory instead of re-reading and
  re-parsing YAML files from disk inside every parallel subtest.
- Optimize the duplicate alias check from O(N^2) to O(N) by
  pre-building a map. This replaces the repeated calls
  to ReportsByAlias which scaled poorly.
Change-Id: I76d9fae0c8dae0dc8d719902aa3e8bbcad2dccb8

Change diff

diff --git a/all_test.go b/all_test.go
index ee095d8..09db6b0 100644
--- a/all_test.go
+++ b/all_test.go
@@ -98,17 +98,32 @@
 		t.Fatal(err)
 	}
 
+	type loadedReport struct {
+		path   string
+		report *report.Report
+	}
+	parsedReports := make(map[string]loadedReport)
+	aliasToIDs := make(map[string][]string)
+	for _, filename := range reports {
+		r, ok := rc.Report(filename)
+		if !ok {
+			t.Fatalf("report %s not found in client", filename)
+		}
+		parsedReports[filename] = loadedReport{path: filename, report: r}
+		for _, alias := range r.Aliases() {
+			aliasToIDs[alias] = append(aliasToIDs[alias], r.ID)
+		}
+	}
+
 	// Map from summaries to report paths, used to check for duplicate summaries.
 	summaries := sync.Map{}
 	sort.Strings(reports)
 	for _, filename := range reports {
 		t.Run(filename, func(t *testing.T) {
-			cFilename := filename // capture variable so we can run tests in parallel safely
+			lr := parsedReports[filename]
+			r := lr.report
+			cFilename := lr.path // capture variable so we can run tests in parallel safely
 			t.Parallel()
-			r, err := report.Read(cFilename)
-			if err != nil {
-				t.Fatal(err)
-			}
 			if err := r.CheckFilename(cFilename); err != nil {
 				t.Error(err)
 			}
@@ -116,17 +131,13 @@
 			if len(lints) > 0 {
 				t.Error(strings.Join(lints, "\n"))
 			}
-			duplicates := make(map[string][]string)
 			for _, alias := range r.Aliases() {
-				for _, r2 := range rc.ReportsByAlias(alias) {
-					if r2.ID != r.ID {
-						duplicates[r2.ID] = append(duplicates[r2.ID], alias)
+				for _, id := range aliasToIDs[alias] {
+					if id != r.ID {
+						t.Errorf("report %s shares duplicate alias %s with report %s", cFilename, alias, id)
 					}
 				}
 			}
-			for r2, aliases := range duplicates {
-				t.Errorf("report %s shares duplicate alias(es) %s with report %s", cFilename, aliases, r2)
-			}
 			// Ensure that each reviewed report has a unique summary.
 			if r.IsReviewed() {
 				if summary := r.Summary.String(); summary != "" {

Change information

Files:
  • M all_test.go
Change size: S
Delta: 1 file changed, 23 insertions(+), 12 deletions(-)
Open in Gerrit

Related details

Attention set is empty
Submit Requirements:
  • requirement is not satisfiedCode-Review
  • requirement is not satisfiedLUCI-Pass
  • requirement satisfiedNo-Unresolved-Comments
  • requirement is not satisfiedReview-Enforcement
  • requirement is not satisfiedTryBots-Pass
Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. DiffyGerrit
Gerrit-MessageType: newchange
Gerrit-Project: vulndb
Gerrit-Branch: master
Gerrit-Change-Id: I76d9fae0c8dae0dc8d719902aa3e8bbcad2dccb8
Gerrit-Change-Number: 732800
Gerrit-PatchSet: 1
Gerrit-Owner: Ethan Lee <etha...@google.com>
Gerrit-Reviewer: Ethan Lee <etha...@google.com>
unsatisfied_requirement
satisfied_requirement
open
diffy

Ethan Lee (Gerrit)

unread,
Dec 26, 2025, 1:47:04 PM (2 days ago) Dec 26
to goph...@pubsubhelper.golang.org, golang-co...@googlegroups.com
Attention needed from Ethan Lee

Ethan Lee uploaded new patchset

Ethan Lee uploaded patch set #2 to this change.
Open in Gerrit

Related details

Attention is currently required from:
  • Ethan Lee
Submit Requirements:
  • requirement is not satisfiedCode-Review
  • requirement is not satisfiedLUCI-Pass
  • requirement satisfiedNo-Unresolved-Comments
  • requirement is not satisfiedReview-Enforcement
  • requirement is not satisfiedTryBots-Pass
Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. DiffyGerrit
Gerrit-MessageType: newpatchset
Gerrit-Project: vulndb
Gerrit-Branch: master
Gerrit-Change-Id: I76d9fae0c8dae0dc8d719902aa3e8bbcad2dccb8
Gerrit-Change-Number: 732800
Gerrit-PatchSet: 2
Gerrit-Owner: Ethan Lee <etha...@google.com>
Gerrit-Reviewer: Ethan Lee <etha...@google.com>
Gerrit-Attention: Ethan Lee <etha...@google.com>
unsatisfied_requirement
satisfied_requirement
open
diffy

Ethan Lee (Gerrit)

unread,
Dec 26, 2025, 1:48:50 PM (2 days ago) Dec 26
to goph...@pubsubhelper.golang.org, Roland Shoemaker, Go LUCI, golang-co...@googlegroups.com
Attention needed from Roland Shoemaker

Ethan Lee voted

Auto-Submit+1
Commit-Queue+1
Open in Gerrit

Related details

Attention is currently required from:
  • Roland Shoemaker
Submit Requirements:
  • requirement is not satisfiedCode-Review
  • requirement is not satisfiedLUCI-Pass
  • requirement satisfiedNo-Unresolved-Comments
  • requirement is not satisfiedReview-Enforcement
  • requirement is not satisfiedTryBots-Pass
Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. DiffyGerrit
Gerrit-MessageType: comment
Gerrit-Project: vulndb
Gerrit-Branch: master
Gerrit-Change-Id: I76d9fae0c8dae0dc8d719902aa3e8bbcad2dccb8
Gerrit-Change-Number: 732800
Gerrit-PatchSet: 2
Gerrit-Owner: Ethan Lee <etha...@google.com>
Gerrit-Reviewer: Ethan Lee <etha...@google.com>
Gerrit-Reviewer: Roland Shoemaker <rol...@golang.org>
Gerrit-Attention: Roland Shoemaker <rol...@golang.org>
Gerrit-Comment-Date: Fri, 26 Dec 2025 18:48:46 +0000
Gerrit-HasComments: No
Gerrit-Has-Labels: Yes
unsatisfied_requirement
satisfied_requirement
open
diffy

Roland Shoemaker (Gerrit)

unread,
Dec 26, 2025, 9:16:28 PM (2 days ago) Dec 26
to Ethan Lee, goph...@pubsubhelper.golang.org, Go LUCI, golang-co...@googlegroups.com
Attention needed from Ethan Lee

Roland Shoemaker voted Code-Review+2

Code-Review+2
Open in Gerrit

Related details

Attention is currently required from:
  • Ethan Lee
Submit Requirements:
  • requirement satisfiedCode-Review
  • requirement satisfiedNo-Unresolved-Comments
  • requirement satisfiedReview-Enforcement
  • requirement satisfiedTryBots-Pass
Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. DiffyGerrit
Gerrit-MessageType: comment
Gerrit-Project: vulndb
Gerrit-Branch: master
Gerrit-Change-Id: I76d9fae0c8dae0dc8d719902aa3e8bbcad2dccb8
Gerrit-Change-Number: 732800
Gerrit-PatchSet: 2
Gerrit-Owner: Ethan Lee <etha...@google.com>
Gerrit-Reviewer: Ethan Lee <etha...@google.com>
Gerrit-Reviewer: Roland Shoemaker <rol...@golang.org>
Gerrit-Attention: Ethan Lee <etha...@google.com>
Gerrit-Comment-Date: Sat, 27 Dec 2025 02:16:24 +0000
Gerrit-HasComments: No
Gerrit-Has-Labels: Yes
satisfied_requirement
open
diffy

Gopher Robot (Gerrit)

unread,
Dec 26, 2025, 9:17:56 PM (2 days ago) Dec 26
to Ethan Lee, goph...@pubsubhelper.golang.org, golang-...@googlegroups.com, Roland Shoemaker, Go LUCI, golang-co...@googlegroups.com

Gopher Robot submitted the change

Change information

Commit message:
all: remove O(N^2) alias check and redundant disk I/O


TestLintReports was hitting the 10-minute timeout in CI due to scaling
bottlenecks as the vulnerability database grows. This change
optimizes the test to scale linearly with the number of reports.

- Eliminate redundant disk I/O by retrieving reports directly from
  the report.LocalClient's memory instead of re-reading and
  re-parsing YAML files from disk inside every parallel subtest.
- Optimize the duplicate alias check from O(N^2) to O(N) by
  pre-building a map. This replaces the repeated calls
  to ReportsByAlias which scaled poorly.
Change-Id: I76d9fae0c8dae0dc8d719902aa3e8bbcad2dccb8
Auto-Submit: Ethan Lee <etha...@google.com>
Reviewed-by: Roland Shoemaker <rol...@golang.org>
Files:
  • M all_test.go
Change size: S
Delta: 1 file changed, 23 insertions(+), 12 deletions(-)
Branch: refs/heads/master
Submit Requirements:
  • requirement satisfiedCode-Review: +2 by Roland Shoemaker
  • requirement satisfiedTryBots-Pass: LUCI-TryBot-Result+1 by Go LUCI
 Open in Gerrit
Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings. DiffyGerrit



 Gerrit-MessageType: merged 


 Gerrit-Project: vulndb 


 Gerrit-Branch: master 


 Gerrit-Change-Id: I76d9fae0c8dae0dc8d719902aa3e8bbcad2dccb8 


 Gerrit-Change-Number: 732800 


 Gerrit-PatchSet: 3 


 Gerrit-Owner: Ethan Lee <etha...@google.com> 


 Gerrit-Reviewer: Ethan Lee <etha...@google.com> 



 Gerrit-Reviewer: Gopher Robot <go...@golang.org> 


 Gerrit-Reviewer: Roland Shoemaker <rol...@golang.org> 



open
diffy
satisfied_requirement
Reply all
Reply to author
Forward
0 new messages