Rate limits for https://crt.sh/

[Rob Stradling]

The crt.sh web service has been struggling to cope with peak traffic levels for some time.  Whenever this happens, many users see HTTP 50x responses.  To ensure that everyone has a fair chance of using the service, today we have added some rate limiting.

Requests to the crt.sh web service are now being throttled at 60 requests per IP address per minute.

This is implemented by the following Nginx directives...

limit_req_zone $crtshlimit_key zone=crtsh:10m rate=60r/m sync;

server {

        limit_req zone=crtsh burst=5;

}

...where $crtshlimit_key is basically $binary_remote_addr.

[Anton Drukh]

Thanks for this amazing service, and for keeping it up and running for the benefit of everyone else!

Just wanted to share an observation and see if it can help deal with load / distribute client requests.

I have a service that pokes crt.sh once every 5 seconds with the most lightweight request I could come up with. I am seeing around 300 daily errors indicating high load - 502/503/504 status codes. The interesting thing is the fact that about two thirds of these errors seem to occur at between 7 and 10 minutes after the hour, i.e. 12:07-12:10, 13:07-13:10 and so on. This makes me think that the service is probably seeing a lot of 'cron'-like requests at the hour mark (12:00, 13:00, etc.), and takes a few mins to get overloaded and start shedding requests.

So if you are planning on using crt.sh in a recurring fashion, staying away from the XX:00-XX:10 timeframe will likely increase your success rate.

Hope this helps!

[r...@sectigo.com]

Hi Anton.  Thanks for reporting this "between 7 and 10 minutes after the hour" issue!  I'm confident that it's now been fixed (by https://github.com/crtsh/certwatch_db/commit/b4dcd7101f763fc667678b3b51c47d3c1c02837e).

crt.sh's internal systems run a bunch of hourly cron scripts (see https://github.com/crtsh/certwatch_db/blob/master/jobs/crontab) against the master database, and one of these scripts was holding an exclusive table lock for that time period.  This had a knock-on effect for users because the affected table is accessed to generate https://crt.sh/ certificate pages.

[Anton Drukh]

Hi Rob,

Thanks for looking into my report, I had no clue this could be internal to crtsh itself!

I'll keep an eye on the distribution of timeouts I'm seeing on my end, in case I can make another basic observation :)