How to query the logs?

[Morgan Taylor]

This might seem like a terribly trivial question, but it's not clear at all to me how to query logs by "subject" (i.e. hostname). As far as I can tell from the RFC, it's necessary to get every single entry and then look through them locally, which sounds extremely inefficient. I've also attempted to use the associated github repo, only to encounter baffling errors. Can anyone offer any guidance on this?

Thanks,

Morgan

[Matt Palmer]

On Sun, Aug 13, 2017 at 10:38:13PM -0700, Morgan Taylor wrote:
> This might seem like a terribly trivial question, but it's not clear at all
> to me how to query logs by "subject" (i.e. hostname). As far as I can tell

> from the RFC <https://tools.ietf.org/html/rfc6962>, it's necessary to get

> every single entry and then look through them locally, which sounds
> extremely inefficient. I've also attempted to use the associated github repo

> <https://tools.ietf.org/html/rfc6962#section-4.6>, only to encounter baffling
> errors
> <https://security.stackexchange.com/questions/167429/how-to-use-certificate-tranparency-library>.

> Can anyone offer any guidance on this?

Sure.

In short, logs aren't *supposed* to be queried. They're an append-only data
structure built for verifiability. Running a log, as specified, is a large
enough undertaking; requiring comprehensive querying capability would make
it even harder to run a log.

The component of the CT ecosystem you're looking for is called a "monitor".
That downloads certificates from logs, optionally verifies that the log is,
indeed, behaving itself, and then indexes the certificates in whatever way
is appropriate for whatever sort of querying is desired (which can be a lot
more than just by subject, *or* hostname).

As you say, it's a significant effort to run a monitor. You've got to
download a *lot* of data. In general, you probably want to use an existing,
publicly-available monitor such as https://crt.sh to do all the heavy
lifting for you, and then you can just run your queries. It's only if
you're particularly keen (or have some pretty large-scale or niche
requirements) that you'd look at running your own monitor.

- Matt

[Ben Laurie]

Note that once we have Trillian up and running properly, it will be possible to make verifiable monitors...

--
You received this message because you are subscribed to the Google Groups "certificate-transparency" group.
To unsubscribe from this group and stop receiving emails from it, send an email to certificate-transparency+unsub...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Morgan Taylor]

Ohhh got it, thank you!!

Any chance you know of any monitors that return JSON rather than HTML?

[Topper Bowers]

I just posted a question on this same topic (whoops... I searched for "search" instead of "query"). I also can't find any information on crt.sh licensing (besides the return value).

[Topper Bowers]

Hey Morgan,

Did you ever figure anything out here? I was just watching Halvar Flake at black hat 2017 and I think we can start to implement some higher-level security concepts. Having a queryable log would be a huge bonus.

Topper