Transparency Logs - Access

[Kevin Jorquera]

Hello, 

I'm looking to scrap some logs, but keep getting a 404.

Is there something not working?

import json
import requests

url = 'https://ct.googleapis.com/logs/argon2023/ct/v1/get-entries'
params = {'start': 0, 'end': 1000}

response = requests.get(url, params=params)
entries = response.json()

with open('output.json', 'w') as f:
json.dump(entries, f)

[Philippe Boneff]

Hi Kevin,

I don't know what's causing the 404 you're observing, would you have more details? 

Do you have any trouble accessing https://ct.googleapis.com/logs/argon2023/ct/v1/get-entries?start=0&end=1000?

Cheers,

Philippe

--
You received this message because you are subscribed to the Google Groups "certificate-transparency" group.
To unsubscribe from this group and stop receiving emails from it, send an email to certificate-transp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/certificate-transparency/09fe4e78-6d60-4cee-b3f4-90e4d4c96910n%40googlegroups.com.

[Kevin Jorquera]

Hey,

Thank you for the fast response. I can access that, but I only see an object with leaf_input as keys .

I apologize for my ignorance, but would this be the best way to monitor domains being registered?

I'm basically trying to build a service that uses machine learning to detect similar registered domains.

[Martin Hutchinson]

Hi Kevin, a few tools that may be useful to you:

I've created a PR to make running the ctclone tool even simpler (if you're used to running docker): https://github.com/google/trillian-examples/pull/818. This will clone the contents of the log into a local database. You'll then need to query this database and parse the certificates to do what you want.

Another option that I saw earlier in the week but have no other context on is https://certstream.calidog.io/ which polls logs and extracts domain names. This may be closer to what you're looking for?

Regards,

Martin

To view this discussion on the web visit https://groups.google.com/d/msgid/certificate-transparency/8fdeaea8-e90e-4f65-b1d9-439ab8abefc8n%40googlegroups.com.