Working with CT Logs

[Sergio Garcia]

Hi Guys,

I have been digging the CT logs as a matter of curiosity as it is a very rich OSINT resource and want to share two projects that I created and some notes for others that are also curious go get hands on this data.

At first, my naïve assumption was that it was pretty simple to get the certificates using the CT Log API, some library to parse it and just profit, but the amount of data, rate limits, unicode issues, etc, make this quite a challenge.

The two projects that I created are:

• https://github.com/sergiogarciadev/ctmon: a client for logs that can monitor domain name regex or save certificates in a database
• https://github.com/sergiogarciadev/libz509pq: a rewrite of the libx509pq using zstandard compression to save storage space

Please note that none of those are production quality, but I think someone may think the ideas there interesting to explore in some way.

-

Sergio Garcia

[r...@sectigo.com]

Hi Sergio.  Thanks for sharing!

libz509pq in interesting.  I've thought about doing much the same thing on previous occasions.  In the case of crt.sh/libx509pq though, we're achieving a similar level of compression at the filesystem level by using ZFS.  The certwatch database is currently 24TB, but it's only consuming about 13TB of actual disk space.

[Sergio Garcia]

Hi Rob,

I also use ZFS and checked the disk size and it was almost the same for both tables, I will put a note on the library README about this about this.

Thanks,