New Log list JSON Schema

[Kat Joyce]

Hello!

New JSON Schema

We have news!  We have been working on a new updated JSON schema for public Log lists to use, and we are now ready to share it with the CT community to get your feedback and make any resulting tweaks and changes before starting to use it!  You can find the new schema at https://www.gstatic.com/ct/log_list/v2_beta/log_list_schema.json.

We have also uploaded an example Log list based on the new schema to https://www.gstatic.com/ct/log_list/v2_beta/log_list_example.json.  This list includes only a handful of Logs, and is simply intended as an example of what the format of a Log list using the new schema might look like.

What will the new schema be used for?

The idea is that it will eventually replace the one at https://www.gstatic.com/ct/log_list/log_list_schema.json, and both https://www.gstatic.com/ct/log_list/all_logs_list.json and https://www.gstatic.com/ct/log_list/log_list.json will be re-generated based on the new schema.  However, before that happens we will have both the new and the old versions hosted on gstatic for a period of time, to allow anyone who currently uses these lists to migrate over to the new schema.

In addition to this, we have also discussed this new schema with Apple, as they too want an updated schema that they could use to publish the list of Logs their software uses.

While there are a few tweaks to structure, and some new basic info fields, the biggest addition in the new schema is the introduction of a 'state' field.  We envisage each UA that enforces CT publishing their own log list based on this schema, where the state they attribute to a Log is based on the state of the Log with respect to that UA.  As such, the states currently in the list have been designed to represent a set of generic states that would be applicable across UAs.

The state machine and meanings of the states

                +------------------------+

                |        |

                |        V

PENDING ---> QUALIFIED ---> USABLE ---> FROZEN------+

  |         | | |   | |   |

  |         | | |   | V   |

  |         | +--------|---+---> RETIRED       |

  |         | |           | V

  +-----------+------------+-----------+------> REJECTED

PENDING:  The Log has requested inclusion in the Log list distributor’s trusted Log list, but has not yet been accepted.  A PENDING Log does not count as ‘currently qualified’, and does not count as ‘once qualified’.

QUALIFIED:  The Log has been accepted by the Log list distributor, and added to the CT checking code used by the Log list distributor.  A QUALIFIED Log counts as ‘currently qualified’.

USABLE:  SCTs from the Log can be relied upon from the perspective of the Log list distributor.  A USABLE Log counts as ‘currently qualified’.

FROZEN:  The Log is trusted by the Log list distributor, but is read-only, i.e. has stopped accepting certificate submissions.  A FROZEN Log counts as ‘currently qualified’.

RETIRED:  The Log was trusted by the Log list distributor up until a specific retirement timestamp.  A RETIRED Log counts as ‘once qualified’ if the SCT in question was issued before the retirement timestamp.  A RETIRED Log does not count as ‘currently qualified’.

REJECTED:  The Log is not and will never be trusted by the Log list distributor.  A REJECTED Log does not count as ‘currently qualified’, and does not count as ‘once qualified’.

Keeping these high-level descriptions in mind, each UA may wish to further specify exactly what each state will encompass wrt them (for example, in Chrome, PENDING covers from when an inclusion bug is submitted, to having monitoring started, to passing monitoring, through to actually being added to the Chrome codebase).  We welcome feedback about potential modifications to these descriptions and any points that could be made clearer!

What would a typical log lifecycle look like?

A normal Log lifecycle:

PENDING --> QUALIFIED --> USABLE --> RETIRED

PENDING --> QUALIFIED --> USABLE --> FROZEN --> RETIRED

A Log fails to meet policy requirements after being accepted by the Log list distributor:

PENDING --> QUALIFIED --> RETIRED

PENDING --> QUALIFIED --> USABLE --> RETIRED

PENDING --> QUALIFIED --> USABLE --> FROZEN --> RETIRED

A Log fails to meet policy requirements before being accepted by the Log list distributor:

PENDING --> REJECTED

(this list is not exhaustive)

What do these states mean for other actors in the CT ecosystem?

CAs: Do not embed or otherwise rely upon SCTs from a Log until it is USABLE.

Monitors: Monitoring should begin, and is valuable, as soon as a Log is PENDING.

What do we need from you, the CT community?

We would really appreciate it if you could go and take a look at the new schema, and let us know what you think! Let the discussion begin!

[Kat Joyce]

Hi Matt,

Thank you for your questions!  Responses inline below.

Kat

On Tue, Oct 23, 2018 at 4:06 PM <matt....@babylonhealth.com> wrote:

Hi Kat,

Just been looking at this new schema as I'm implementing certificate transparency on Android.

So far I have a couple of questions regarding "state".

Firstly, should this be marked as "required" in the schema? Wasn't sure with jsonschema if 'oneOf' indirectly makes this required.

Not quite - the oneOf that's in there makes it so that *if* the state field is present, it has to have one of the actual state fields (pending, qualified, ... etc) specified within it.  This does mean that the state field could be left out entirely, and still adhere to the JSON schema.  This was intended to allow flexibility in the use of the schema.  The states themselves are typically going to be specific to a given UA, and I mentioned in the email that one of the ways we plan to use this schema is to generate a new 'all_logs_list.json' equivalent.  This is intended to be a list of all the CT Logs we are aware of, some of which have never applied for inclusion in Chrome, and so don't have a state from Chrome's perspective.  These Logs will not have the state field present in their definition in the new all_logs_list.json.

 

Secondly, from a user agents perspective when verifying an SCT would we then do the following:

    PENDING, REJECTED -> An SCT associated with this log server would be treated as untrusted

    QUALIFIED, USABLE, FROZEN -> Validate SCT against this (any timestamp okay)

    RETIRED -> Validate SCT against this if it was issued before the state timestamp, otherwise SCT is untrusted

That looks about right to me :)

That being said, the way that Chrome does it is to have the list of Logs they trust built in to their source code, and they only really have two states within their built in list - regular or retired (indicated by the presence of a timestamp at which the Log was retired).  The three buckets that an SCT can fall in to then become 'not in the built in list', 'in the list with no timestamp' and 'in the list with a timestamp', which correspond to the three categories you describe above.  This information may not be of use to you, but I thought I'd mention how Chrome currently does it, just in case it is :)

 

Also, what is the timeframe for rolling out the new schema?

We are going to be presenting, discussing and getting feedback on the new schema at the upcoming CT Policy days hosted by Apple on Nov 6th/7th.  Around a similar time we also hope to publish our log lists based on this schema, also within the v2_beta directory, alongside where the schema currently resides.  Once all of that is done, and any feedback has been incorporated, we will let everyone know, and encourage people to start using the new schema.  We will have both schemas hosted on gstatic for a period of time, to allow people to switch from using one to the other, and we will send out notice with plenty of warning before taking down the old schema.  The exact timeline of this transitional period is still TBD, and will be influenced by feedback from the community and what people need.

 

Many thanks,

Matt Dolan

--
You received this message because you are subscribed to the Google Groups "certificate-transparency" group.
To unsubscribe from this group and stop receiving emails from it, send an email to certificate-transp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/certificate-transparency/47b76ebf-7485-4e15-82c2-b58f5f8add92%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Kat Joyce]

Hi everyone,

A quick update on the new Log list JSON schema:

After all of the feedback we received at the CT policy days in November, we have been working with interested parties to make some changes to the schema, which can now be seen at https://www.gstatic.com/ct/log_list/v2_beta/log_list_schema.json, with a dummy example list based on this schema at https://www.gstatic.com/ct/log_list/v2_beta/log_list_example.json.

Specifically, the changes that were made are:

  - to change the 'operators' and 'logs' fields to arrays (they were previously maps/dictionaries)

  - to change 'description' from an array to a string

  - to add top-level 'version' field

  - to re-name 'frozen' to 'readonly'

If anyone has any more feedback on the schema, please do get in touch!  We'll keep everyone informed of finalization plans when we get there :)

Thanks!

Kat

[Bailey Basile]

Hello, all!

We have updated our log list schema and log list to be compliant with the new beta schema that Kat has posted.

Apple's current shipping log list is always available at https://valid.apple.com/ct/log_list/current_log_list.json.

Our current log list schema is always available at https://valid.apple.com/ct/log_list/current_log_list_schema.json.

Prior versions of our log list may be found under https://valid.apple.com/ct/log_list/log_list_versions/log_list_v<log list version>.json (e.g. https://valid.apple.com/ct/log_list/log_list_versions/log_list_v22.json).

We don't currently have any old versions posted publicly before version 19.

Bailey

To unsubscribe from this group and stop receiving emails from it, send an email to certificate-transparency+unsub...@googlegroups.com.

[Kat Joyce]

Hi everyone,

TL:DR; New log lists are now available under https://www.gstatic.com/ct/log_list/v2/, switch to these by September 30th as https://www.gstatic.com/ct/log_list/v2_beta/* will be removed then.

We are excited to announce that having finalized the new Log list schema, we are now publishing our log lists using it.

The new schema and log lists can all be found under https://www.gstatic.com/ct/log_list/v2/

Specifically, the following are now available:

https://www.gstatic.com/ct/log_list/v2/log_list_schema.json - the new JSON schema.

https://www.gstatic.com/ct/log_list/v2/all_logs_list.json - the log list containing all known Logs (past and present).

https://www.gstatic.com/ct/log_list/v2/log_list.json - the list of Logs that are currently compliance with Chrome's CT Log policy (or have been and were retires), and are included in Chrome.

https://www.gstatic.com/ct/log_list/v2/log_list.sig - Google's signature for log_list.json

https://www.gstatic.com/ct/log_list/v2/log_list_pubkey.pem - the public key required to verify log_list.sig

We will be updating the page at https://www.certificate-transparency.org/known-logs shortly to reflect this.

Some of you will be aware that we were serving these lists from a v2_beta directory previous to this announcement.  This is notice that on September 30th 2019 https://www.gstatic.com/ct/log_list/v2_beta/* will be removed.  Therefore, you should switch over to using the lists hosted at https://www.gstatic.com/ct/log_list/v2/ as soon as possible.

Do let us know if you find any issues with these new lists!  Otherwise, we hope they are of value to you!

Kat

On Sun, Aug 11, 2019 at 12:49 PM Matthew Dolan <matt....@babylonhealth.com> wrote:

Hi Kat,

Just wondering what the status is of the v2 schema. Is this `2.0.beta2` version finalised and when will it likely be adopted?

Thanks,

Matt

On Friday, January 4, 2019 at 4:48:29 PM UTC, Kat Joyce wrote:

Hi everyone,

A quick update on the new Log list JSON schema:

After all of the feedback we received at the CT policy days in November, we have been working with interested parties to make some changes to the schema, which can now be seen at https://www.gstatic.com/ct/log_list/v2_beta/log_list_schema.json, with a dummy example list based on this schema at https://www.gstatic.com/ct/log_list/v2_beta/log_list_example.json.

Specifically, the changes that were made are:

  - to change the 'operators' and 'logs' fields to arrays (they were previously maps/dictionaries)

  - to change 'description' from an array to a string

  - to add top-level 'version' field

  - to re-name 'frozen' to 'readonly'

If anyone has any more feedback on the schema, please do get in touch!  We'll keep everyone informed of finalization plans when we get there :)

Thanks!

Kat

On Tue, Oct 23, 2018 at 4:42 PM Kat Joyce <katj...@google.com> wrote:

Hi Matt,

Thank you for your questions!  Responses inline below.

Kat

To view this discussion on the web visit https://groups.google.com/d/msgid/certificate-transparency/e2710a80-3710-4acf-b208-decf1c371fff%40googlegroups.com.