Crt.sh seems to have great problems

[Sebastian Nielsen]

crt.sh seems to have some problems. It issues gateway error randomly half of the requests, and also my certificate with serial 0394eb48c7e1cf5f79f119689dc931e8a9c9 (fingerprint: 9f7bd524588ac785b796ba9c3f68bd6d28a62eb) isn't there suggesting the actual CT log server might have some weird difficulties too.

[Rob Stradling]

Hi Sebastian.

We believe that the frequent gateway errors are due to the dodgy SSDs that the front-end crt.sh servers are currently using.  New SSDs were ordered a while ago and are due to be delivered this week, but since we're all working from home I don't think any Sectigo staff will be available to receive the delivery.  But anyway, at some point in the future we'll be able to put the new SSDs into service, after which we anticipate that https://crt.sh and crt.sh:5432 should perform much better.

Your certificate is missing because crt.sh currently has some large backlogs (see https://crt.sh/monitored-logs).  These backlogs have been caused by some index rebuilds that I kicked off on the crt.sh master database nearly a week ago.  I estimate that the index rebuilds should complete sometime over this coming weekend, after which the backlogs will start to go down again.  It will probably take several weeks for the backlogs to drop to zero.

The index rebuilds are necessary to support improvements to wildcard searching, which a number of users have requested (both on this forum and privately).

Sorry for the inconvenience.  Running a CT log aggregator is hard!

[Rob Stradling]

The index rebuilds finished a couple of days ago, and the backlogs (see https://crt.sh/monitored-logs) have been going down since then.

As a temporary measure (until we are able to accept delivery of the new SSDs and put them into service) that will (hopefully!) improve performance, our ops team have moved the front-end slave databases onto a different storage array.