crt.sh slow and unusable for the past 24 hours - 502 Bad Gateway errors

[Sam Stepanyan]

Getting 502 Bad Gateway errors or just timeouts for the past 24 hours  - anyone else experiencing the same?

[Rob Stradling]

Hi Sam.  Has the service been more usable from your perspective over the last few days?

I'd kicked off a very-long-running query on the master database the day before your post.  Killing that query seemed to help improve the usability of the service, although I have no clue why this would've helped.

The front-end servers for the "new" crt.sh service are currently still being swamped at times (often resulting in 502s, I think), despite the rate limits we introduced last week.  We believe the core problem is that the front-end servers are currently using a bunch of SSDs that were rejected by other Sectigo projects as being unfit for purpose due to poor performance.  Although we had hoped that these SSDs would be good enough for crt.sh's requirements, it has become clear that they're not.  I've put in a request for new SSDs, but I haven't yet heard back from our Ops and Finance folks.