Feature request: Easier update detection for ccadb data (hash/metadata)

[Hanno Böck]

Hi,

I would like to make a suggestion to make the use of CCADB data easier.

Right now, there does not appear to be any easy way to track updates
of the various CSVs (e.g., AllCertificateRecordsCSVFormatv2). If one
wants to use the data, essentially the only way to keep it updated is
to re-download it regularly.

The server responds rather slowly (is this data live-generated?), and
the "last-modified" header does not appear to contain useful data (it
just responds with the current date/time).

Would it be possible to have some kind of small, static metadata file
with a hash and possibly latest update time of all CSVs?
It could be something as simple as
AllCertificateRecordsCSVFormatv2,[timestamp],[sha256]
AllIncludedRootCertsCSV,...

That would allow just fetching that small file to check for updates.

--
Hanno Böck - Independent security researcher
https://itsec.hboeck.de/
https://badkeys.info/

[Chris Clements]

Hi Hanno, 

Thank you for the suggestion.

The server responds rather slowly (is this data live-generated?), and
the "last-modified" header does not appear to contain useful data (it
just responds with the current date/time).

The data is indeed generated in real time. The resulting CSV contains the most up-to-date information from the CCADB. 

Would it be possible to have some kind of small, static metadata file
with a hash and possibly latest update time of all CSVs?
It could be something as simple as
AllCertificateRecordsCSVFormatv2,[timestamp],[sha256]
AllIncludedRootCertsCSV,...

Knowing that the data is live-generated, here are some other things we could consider in short order:

• Add a column showing the date/time when the report was generated.
• Update the report to generate a CSV file with a dynamic name that includes the current date/time (e.g., AllCertificateRecordsCSVFormatv2_10_20_2024_23:35.csv).
• Create a separate (smaller) report that includes the records changed within the last 24 or 48 hours (or any other time frame).

Do any of these sound appealing?

Thanks again!
-Chris, on behalf of the CCADB Steering Committee

--
You received this message because you are subscribed to the Google Groups "CCADB Public" group.
To unsubscribe from this group and stop receiving emails from it, send an email to public+un...@ccadb.org.
To view this discussion on the web visit https://groups.google.com/a/ccadb.org/d/msgid/public/20241019133840.40ad8d74%40computer.

[Hanno Böck]

Hi,

On Mon, 21 Oct 2024 16:38:16 -0400
"'Chris Clements' via CCADB Public" <pub...@ccadb.org> wrote:

> Knowing that the data is live-generated, here are some other things we
> could consider in short order:
>

> - Add a column showing the date/time when the report was generated.

It would appear that this would make it more difficult to detect
neccessity to update (as each report would have a different hash, even
"download and compare the file content" would not work).

> - Update the report to generate a CSV file with a dynamic name that

> includes the current date/time (e.g.,
> AllCertificateRecordsCSVFormatv2_10_20_2024_23:35.csv).

> - Create a separate (smaller) report that includes the records

> changed within the last 24 or 48 hours (or any other time frame).
>
> Do any of these sound appealing?

Not really.

Just to reiterate what I would prefer to have:
Assume I have a copy of some of the CSV files locally, and I want a
quick way (aka faster than complete re-download) to check if it's
up-to-date.

If that's not possible, full download it is (but please don't make
changes that make it even harder).

[Chris Clements]

Just to reiterate what I would prefer to have:
Assume I have a copy of some of the CSV files locally, and I want a
quick way (aka faster than complete re-download) to check if it's
up-to-date.

Unfortunately, we do not have a quick fix for this today. 

However, the CCADB SC has recently started discussing a desire to replicate the CCADB to enhance data availability and allow for self-serve/public access by security researchers and other interested parties. We intend to initiate a separate discussion here in pub...@ccadb.org to collect some public opinions on solutioning. More to come!