Fair Usage

216 views

Skip to first unread message

Mark Alford

unread,

Oct 19, 2024, 6:03:43 PM10/19/24

to crt.sh

Hey,

Query on fair usage - I have 100,000 to 1m domains I need to query, and don't want to overload system. Do you have any guidelines on fair usage?

Would likely be looking at web scraping as I've found the database to be extremely slow (40s vs <1s for simple domain lookup).

I was thinking just perform requests sequentially, so not hammering in parallel and overloading, seems to be doing 1 every few seconds which seems to be inline with database suggestions of 2 connections per second - is that ok?

BTW great resource, thanks for your effort.

Best regards

r...@sectigo.com

unread,

Oct 30, 2024, 8:49:33 AM10/30/24

to crt.sh

Hi Mark. I just discovered that pg_bouncer had crashed (again) on some of the crt.sh:5432 servers. I've fixed that, so hopefully crt.sh:5432 is a bit quicker now.

> I was thinking just perform requests sequentially, so not hammering in parallel and overloading, seems to be doing 1 every few seconds which seems to be inline with database suggestions of 2 connections per second - is that ok?