Poor Availability of Sectigo Logs

[Andrew Ayer]

For more than a month, several certspotter users have reported growing backlogs when monitoring Sectigo logs, along with a high rate of errors, such as 500 internal server errors, 429 too many requests errors, and full Cloudflare "rendering error" HTML pages.

crt.sh reports backlogs of over 100 million for Sabre 2025h2 and Mammoth 2025h2.

Chrome's availability monitoring reports uptimes of less than 90% for Sabre 2025h2 and Mammoth 2025h2.

SSLMate has been able to avoid backlogs by sending requests from a pool of 4 IP addresses, but this should not be required.

I don't think this is acceptable for trusted CT logs, and I have the following questions for Sectigo:

1. Can the rate limits of the write endpoints be reduced as an immediate mitigation for growing backlogs in monitors?

2. Are requests that result in 5XX errors counted against the rate limits?

3. What are the plans for improving the availability of these logs?

Regards,
Andrew

[Rob Stradling]

Hi Andrew.

I realise we've been quiet on this list for some time despite the ongoing availability issues with our logs, and I'd like to apologise to everyone for that.  We have been busy working on a solution, which has taken rather longer than anticipated to put together.  You've started this thread just as we were gearing up to make an announcement.

> 1. Can the rate limits of the write endpoints be reduced as an immediate mitigation for growing backlogs in monitors?

I will discuss this with our Ops team and get back to you.

> 2. Are requests that result in 5XX errors counted against the rate limits?

Yes, currently they are.

> 3. What are the plans for improving the availability of these logs?

Please see my next post (in a new thread).

[Rob Stradling]

> > 1. Can the rate limits of the write endpoints be reduced as an immediate mitigation for growing backlogs in monitors?
>
> I will discuss this with our Ops team and get back to you.

We updated the rate limits for the Mammoth and Sabre shards as follows, about 24 hours ago:

• Removed: Per-IP rate limits.  (This limit had begun to limit Cloudflare's IPs, which wasn't useful.  We may revisit per-IP limits in the future).
• Added: 200 req/sec limit for POST requests, per shard.

Other rate limits that remain in place for these shards:

• 400 req/sec limit for all requests, per shard.
• 40 req/sec limit for POST requests for "old" submissions (notBefore older than 28hrs at time of submission).

Is this mitigation sufficient?  If not, we can further reduce the 400 req/sec limit.

[Rob Stradling]

Hi Winston.

> Is this mitigation sufficient?  If not, we can further reduce the 400 req/sec limit.

Sorry, "400" was a typo there.

I meant to say that, if the mitigation is not sufficient, we can further reduce the 200 req/sec per-shard POST limit.

> It seems to me like a higher read request limit is better.

For the purposes of helping monitors chew through ingestion backlogs, yes, I agree.

---

From: Winston de Greef <winston...@gmail.com>
Sent: 10 July 2025 19:41
To: Rob Stradling <r...@sectigo.com>
Cc: Certificate Transparency Policy <ct-p...@chromium.org>; Andrew Ayer <ag...@andrewayer.name>
Subject: Re: [ct-policy] Re: Poor Availability of Sectigo Logs

 

This Message Is From an External Sender

This message came from outside your organization.

Report Suspicious

 

Hi Rob,

Instead of further decreasing the 400 request/s limit, is it possible to increase this limit? It seems to me like a higher read request limit is better.

Sincerely,

Winston de Greef

--
You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+...@chromium.org.
To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/495f546f-3b80-4634-b678-851e8cc79f05n%40chromium.org.

[Winston de Greef]

Hi Rob,

Instead of further decreasing the 400 request/s limit, is it possible to increase this limit? It seems to me like a higher read request limit is better.

Sincerely,

Winston de Greef

On Thu, Jul 10, 2025, 9:35 AM 'Rob Stradling' via Certificate Transparency Policy <ct-p...@chromium.org> wrote:

--