Backlog on monitored logs ?

[Mathieu Goessens]

Hello,

I am observing a >=1 week delay for some certificated to appears on
crt.sh, with only the pre-certificate being available meanwhile.

https://crt.sh/monitored-logs seems to show an important delay and
backlog for some logs from Google, LetsEncrypt, and Digicert, and I am
wondering if it is related.

Thanks a lot for your work,

Regards,

--
Mathieu Goessens
Université de Rennes / ISTIC / EUR Cyberschool

[pieter hartel]

I have noticed something similar:

It seems that the number of OV certificates is dwindling. Here are some statistics:

20241122 25961
20241123 22950
20241124 26121
20241125 31796
20241126 28327
20241127 34579
20241128 fail
20241129 26894
20241130 45158
20241201 27226
20241202 36528
20241203 fail
20241204 19853
20241205 9193
20241206 5610
20241207 1636
20241208 2915
20241209 fail
20241210 7587
20241211 6247
20241212 6289
20241213 4566
20241214 1375

The fail entries are due to network issues, I'm not sure where...

--pieter

[r...@sectigo.com]

Hi.  Apologies for the delay in replying to this thread.

Starting October 30th, we began to experience some degradation of performance on the infrastructure on which crt.sh runs. This resulted in the service falling significantly behind on ingestion of new log entries.

Sectigo remains committed to operating crt.sh for the benefit of the community and at no cost, and we wanted to acknowledge (albeit belatedly) these performance issues and assure everyone that we've been taking steps to resolve them:

• Temporary measure: We have disabled log ingestion from the Sycamore/Willow "Static CT" logs, which are not (yet) Qualified or Usable in any CT clients.  (Once crt.sh has caught up on the ingestion backlog for the other logs, we will resume ingesting from Sycamore/Willow).
• Short-term measures:
  • We have adjusted the rate-limiting for direct database access on crt.sh:5432 to allow for fairer use of the available resources.
  • We have offloaded the postgres pg_wal directories to a different storage system.
  • We have offloaded postgres temporary files to a tmpfs RAM disk.

Thanks to these short-term measures, crt.sh is now ingesting log entries fast enough to be catching up on the ingestion backlog at a rate of approximately 1 million entries per hour.  Assuming that rate of progress continues, it should be able to chew through the current ingestion backlog of 760 million over the next month or so.

We are also looking at various potential longer-term changes to future-proof crt.sh's performance.  These include hardware upgrades and a possible move of some or all crt.sh components to a cloud provider.

[r...@sectigo.com]

> Assuming that rate of progress continues, it should be able to chew through the current ingestion backlog of 760 million over the next month or so.

Unfortunately we've run into more infrastructure performance issues over the past week or so, meaning that we definitely won't achieve that projected target.

[r...@sectigo.com]

Due to further infrastructure performance issues over the past month, there is still a large ingestion backlog at this point in time.

A newer, faster storage array was delivered recently to the data centre from where crt.sh now runs.  Our Ops team is planning to configure and move crt.sh's filesystems on to this device in the coming days.  We expect that this will resolve the performance issues that have plagued crt.sh in recent months.

We are still looking at various other potential longer-term changes to future-proof crt.sh's performance, including a possible move of some or all crt.sh components to a cloud provider.  However, for now, our focus is on stabilising the service in its current form.

[r...@sectigo.com]

Our Ops team finished migrating all of the crt.sh services over to the newer, faster storage array just over a week ago.  Everything seems to be performing much better.

The log entry ingestion backlog is going down.  I'm estimating that the backlog will disappear in the next 2 to 4 weeks.

[r...@sectigo.com]

The log entry ingestion backlog, which began in October 2024, finally disappeared a couple of weeks ago as anticipated.

The only current ingestion backlogs are due to certain logs rate-limiting /ct/v1/get-entries calls to the extent that monitors are unable to retrieve entries quickly enough using a single IP address.