Low add-chain availability on Nimbus2026 log

[Luke Valenta]

Hi folks,

Between 2025-10-03 01:00 UTC and 2025-10-06 6:30 UTC, Cloudflare’s Nimbus2026 log was returning high rates of 403s from the add-chain and add-pre-chain endpoints due to an increased backlog. Upon investigation, we found an increase in request volume from a small number of IPs that coincided with the start of impact. The increased log volume can be clearly seen at https://radar.cloudflare.com/certificate-transparency/log/nimbus2026?dateStart=2025-10-01&dateEnd=2025-10-06.

A similar occurrence happened to the Nimbus2025 log shard in March 2025 (https://issues.chromium.org/issues/40927575#comment9), at which time we instituted a 

10 rps per IP for the add-chain and add-pre-chain endpoints, which mitigated impact. However, that rate limit was specific to the Nimbus2025 log shard and thus did not automatically mitigate the current issue.

After adjusting the rate limiting rule to cover all Nimbus log shards, the Nimbus2026 log shard is no longer returning 403s and the impact appears to be over.

We'd like to thank the Chrome team for notifying us of the issue. This revealed a gap in our alerting, as our engineering team was not automatically notified when the logs started returning 403s. We've added some improved alerting notifications in the short term, and are investing in better monitoring and alerting for our CT logs (Nimbus and Raio) in the medium term.

Best,
Luke

--

Luke Valenta

Systems Engineer - Research