New Log list JSON Schema

763 views
Skip to first unread message

Kat Joyce

unread,
Oct 1, 2018, 10:16:19 AM10/1/18
to certificate-...@googlegroups.com, Certificate Transparency Policy

Hello!


New JSON Schema


We have news!  We have been working on a new updated JSON schema for public Log lists to use, and we are now ready to share it with the CT community to get your feedback and make any resulting tweaks and changes before starting to use it!  You can find the new schema at https://www.gstatic.com/ct/log_list/v2_beta/log_list_schema.json.


We have also uploaded an example Log list based on the new schema to https://www.gstatic.com/ct/log_list/v2_beta/log_list_example.json.  This list includes only a handful of Logs, and is simply intended as an example of what the format of a Log list using the new schema might look like.


What will the new schema be used for?


The idea is that it will eventually replace the one at https://www.gstatic.com/ct/log_list/log_list_schema.json, and both https://www.gstatic.com/ct/log_list/all_logs_list.json and https://www.gstatic.com/ct/log_list/log_list.json will be re-generated based on the new schema.  However, before that happens we will have both the new and the old versions hosted on gstatic for a period of time, to allow anyone who currently uses these lists to migrate over to the new schema.


In addition to this, we have also discussed this new schema with Apple, as they too want an updated schema that they could use to publish the list of Logs their software uses.


While there are a few tweaks to structure, and some new basic info fields, the biggest addition in the new schema is the introduction of a 'state' field.  We envisage each UA that enforces CT publishing their own log list based on this schema, where the state they attribute to a Log is based on the state of the Log with respect to that UA.  As such, the states currently in the list have been designed to represent a set of generic states that would be applicable across UAs.


The state machine and meanings of the states


                +------------------------+

                |        |

                |        V

PENDING ---> QUALIFIED ---> USABLE ---> FROZEN------+

  |         | | |   | |   |

  |         | | |   | V   |

  |         | +--------|---+---> RETIRED       |

  |         | |           | V

  +-----------+------------+-----------+------> REJECTED


PENDING:  The Log has requested inclusion in the Log list distributor’s trusted Log list, but has not yet been accepted.  A PENDING Log does not count as ‘currently qualified’, and does not count as ‘once qualified’.


QUALIFIED:  The Log has been accepted by the Log list distributor, and added to the CT checking code used by the Log list distributor.  A QUALIFIED Log counts as ‘currently qualified’.


USABLE:  SCTs from the Log can be relied upon from the perspective of the Log list distributor.  A USABLE Log counts as ‘currently qualified’.


FROZEN:  The Log is trusted by the Log list distributor, but is read-only, i.e. has stopped accepting certificate submissions.  A FROZEN Log counts as ‘currently qualified’.


RETIRED:  The Log was trusted by the Log list distributor up until a specific retirement timestamp.  A RETIRED Log counts as ‘once qualified’ if the SCT in question was issued before the retirement timestamp.  A RETIRED Log does not count as ‘currently qualified’.


REJECTED:  The Log is not and will never be trusted by the Log list distributor.  A REJECTED Log does not count as ‘currently qualified’, and does not count as ‘once qualified’.


Keeping these high-level descriptions in mind, each UA may wish to further specify exactly what each state will encompass wrt them (for example, in Chrome, PENDING covers from when an inclusion bug is submitted, to having monitoring started, to passing monitoring, through to actually being added to the Chrome codebase).  We welcome feedback about potential modifications to these descriptions and any points that could be made clearer!


What would a typical log lifecycle look like?


A normal Log lifecycle:

PENDING --> QUALIFIED --> USABLE --> RETIRED

PENDING --> QUALIFIED --> USABLE --> FROZEN --> RETIRED


A Log fails to meet policy requirements after being accepted by the Log list distributor:

PENDING --> QUALIFIED --> RETIRED

PENDING --> QUALIFIED --> USABLE --> RETIRED

PENDING --> QUALIFIED --> USABLE --> FROZEN --> RETIRED


A Log fails to meet policy requirements before being accepted by the Log list distributor:

PENDING --> REJECTED


(this list is not exhaustive)


What do these states mean for other actors in the CT ecosystem?


CAs: Do not embed or otherwise rely upon SCTs from a Log until it is USABLE.

Monitors: Monitoring should begin, and is valuable, as soon as a Log is PENDING.


What do we need from you, the CT community?


We would really appreciate it if you could go and take a look at the new schema, and let us know what you think! Let the discussion begin!

matt....@babylonhealth.com

unread,
Oct 23, 2018, 11:06:04 AM10/23/18
to certificate-transparency
Hi Kat,

Just been looking at this new schema as I'm implementing certificate transparency on Android.

So far I have a couple of questions regarding "state".

Firstly, should this be marked as "required" in the schema? Wasn't sure with jsonschema if 'oneOf' indirectly makes this required.

Secondly, from a user agents perspective when verifying an SCT would we then do the following:

    PENDING, REJECTED -> An SCT associated with this log server would be treated as untrusted

    QUALIFIED, USABLE, FROZEN -> Validate SCT against this (any timestamp okay)

    RETIRED -> Validate SCT against this if it was issued before the state timestamp, otherwise SCT is untrusted


Also, what is the timeframe for rolling out the new schema?


Many thanks,

Matt Dolan

Kat Joyce

unread,
Oct 23, 2018, 11:43:24 AM10/23/18
to matt....@babylonhealth.com, ct-p...@chromium.org, certificate-...@googlegroups.com
Hi Matt,

Thank you for your questions!  Responses inline below.

Kat

On Tue, Oct 23, 2018 at 4:06 PM <matt....@babylonhealth.com> wrote:
Hi Kat,

Just been looking at this new schema as I'm implementing certificate transparency on Android.

So far I have a couple of questions regarding "state".

Firstly, should this be marked as "required" in the schema? Wasn't sure with jsonschema if 'oneOf' indirectly makes this required.

Not quite - the oneOf that's in there makes it so that *if* the state field is present, it has to have one of the actual state fields (pending, qualified, ... etc) specified within it.  This does mean that the state field could be left out entirely, and still adhere to the JSON schema.  This was intended to allow flexibility in the use of the schema.  The states themselves are typically going to be specific to a given UA, and I mentioned in the email that one of the ways we plan to use this schema is to generate a new 'all_logs_list.json' equivalent.  This is intended to be a list of all the CT Logs we are aware of, some of which have never applied for inclusion in Chrome, and so don't have a state from Chrome's perspective.  These Logs will not have the state field present in their definition in the new all_logs_list.json.
 

Secondly, from a user agents perspective when verifying an SCT would we then do the following:

    PENDING, REJECTED -> An SCT associated with this log server would be treated as untrusted

    QUALIFIED, USABLE, FROZEN -> Validate SCT against this (any timestamp okay)

    RETIRED -> Validate SCT against this if it was issued before the state timestamp, otherwise SCT is untrusted

That looks about right to me :)

That being said, the way that Chrome does it is to have the list of Logs they trust built in to their source code, and they only really have two states within their built in list - regular or retired (indicated by the presence of a timestamp at which the Log was retired).  The three buckets that an SCT can fall in to then become 'not in the built in list', 'in the list with no timestamp' and 'in the list with a timestamp', which correspond to the three categories you describe above.  This information may not be of use to you, but I thought I'd mention how Chrome currently does it, just in case it is :)
 
Also, what is the timeframe for rolling out the new schema?

We are going to be presenting, discussing and getting feedback on the new schema at the upcoming CT Policy days hosted by Apple on Nov 6th/7th.  Around a similar time we also hope to publish our log lists based on this schema, also within the v2_beta directory, alongside where the schema currently resides.  Once all of that is done, and any feedback has been incorporated, we will let everyone know, and encourage people to start using the new schema.  We will have both schemas hosted on gstatic for a period of time, to allow people to switch from using one to the other, and we will send out notice with plenty of warning before taking down the old schema.  The exact timeline of this transitional period is still TBD, and will be influenced by feedback from the community and what people need.
 
--
You received this message because you are subscribed to the Google Groups "certificate-transparency" group.
To unsubscribe from this group and stop receiving emails from it, send an email to certificate-transp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/certificate-transparency/47b76ebf-7485-4e15-82c2-b58f5f8add92%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Kat Joyce

unread,
Jan 4, 2019, 11:48:29 AM1/4/19
to matt....@babylonhealth.com, Certificate Transparency Policy, certificate-...@googlegroups.com
Hi everyone,

A quick update on the new Log list JSON schema:

After all of the feedback we received at the CT policy days in November, we have been working with interested parties to make some changes to the schema, which can now be seen at https://www.gstatic.com/ct/log_list/v2_beta/log_list_schema.json, with a dummy example list based on this schema at https://www.gstatic.com/ct/log_list/v2_beta/log_list_example.json.

Specifically, the changes that were made are:
  - to change the 'operators' and 'logs' fields to arrays (they were previously maps/dictionaries)
  - to change 'description' from an array to a string
  - to add top-level 'version' field
  - to re-name 'frozen' to 'readonly'

If anyone has any more feedback on the schema, please do get in touch!  We'll keep everyone informed of finalization plans when we get there :)

Thanks!
Kat

Matthew Dolan

unread,
Aug 11, 2019, 7:49:47 AM8/11/19
to certificate-transparency
Hi Kat,

Just wondering what the status is of the v2 schema. Is this `2.0.beta2` version finalised and when will it likely be adopted?


Thanks,

Matt

On Friday, January 4, 2019 at 4:48:29 PM UTC, Kat Joyce wrote:
Hi everyone,

A quick update on the new Log list JSON schema:

After all of the feedback we received at the CT policy days in November, we have been working with interested parties to make some changes to the schema, which can now be seen at https://www.gstatic.com/ct/log_list/v2_beta/log_list_schema.json, with a dummy example list based on this schema at https://www.gstatic.com/ct/log_list/v2_beta/log_list_example.json.

Specifically, the changes that were made are:
  - to change the 'operators' and 'logs' fields to arrays (they were previously maps/dictionaries)
  - to change 'description' from an array to a string
  - to add top-level 'version' field
  - to re-name 'frozen' to 'readonly'

If anyone has any more feedback on the schema, please do get in touch!  We'll keep everyone informed of finalization plans when we get there :)

Thanks!
Kat

On Tue, Oct 23, 2018 at 4:42 PM Kat Joyce <katj...@google.com> wrote:
Hi Matt,

Thank you for your questions!  Responses inline below.

Kat

To unsubscribe from this group and stop receiving emails from it, send an email to certificate-transparency+unsub...@googlegroups.com.

Kat Joyce

unread,
Aug 16, 2019, 8:31:02 AM8/16/19
to certificate-...@googlegroups.com, Certificate Transparency Policy
Hi everyone,

TL:DR; New log lists are now available under https://www.gstatic.com/ct/log_list/v2/, switch to these by September 30th as https://www.gstatic.com/ct/log_list/v2_beta/* will be removed then.

We are excited to announce that having finalized the new Log list schema, we are now publishing our log lists using it.

The new schema and log lists can all be found under https://www.gstatic.com/ct/log_list/v2/

Specifically, the following are now available:

https://www.gstatic.com/ct/log_list/v2/all_logs_list.json - the log list containing all known Logs (past and present).
https://www.gstatic.com/ct/log_list/v2/log_list.json - the list of Logs that are currently compliance with Chrome's CT Log policy (or have been and were retires), and are included in Chrome.
https://www.gstatic.com/ct/log_list/v2/log_list.sig - Google's signature for log_list.json
https://www.gstatic.com/ct/log_list/v2/log_list_pubkey.pem - the public key required to verify log_list.sig

We will be updating the page at https://www.certificate-transparency.org/known-logs shortly to reflect this.

Some of you will be aware that we were serving these lists from a v2_beta directory previous to this announcement.  This is notice that on September 30th 2019 https://www.gstatic.com/ct/log_list/v2_beta/* will be removed.  Therefore, you should switch over to using the lists hosted at https://www.gstatic.com/ct/log_list/v2/ as soon as possible.

Do let us know if you find any issues with these new lists!  Otherwise, we hope they are of value to you!

Kat


To unsubscribe from this group and stop receiving emails from it, send an email to certificate-transp...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "certificate-transparency" group.
To unsubscribe from this group and stop receiving emails from it, send an email to certificate-transp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/certificate-transparency/e2710a80-3710-4acf-b208-decf1c371fff%40googlegroups.com.

SriRamji Kathiravan

unread,
Aug 22, 2019, 9:43:30 AM8/22/19
to certificate-transparency
Hi Kat Joyce

Are we still supporting version one log list

We are not able verify the signature for below log_list 

Exception :  InValid Signature


cc: Matthew Dolan


On Friday, August 16, 2019 at 6:01:02 PM UTC+5:30, Kat Joyce wrote:
Hi everyone,

TL:DR; New log lists are now available under https://www.gstatic.com/ct/log_list/v2/, switch to these by September 30th as https://www.gstatic.com/ct/log_list/v2_beta/* will be removed then.

We are excited to announce that having finalized the new Log list schema, we are now publishing our log lists using it.

The new schema and log lists can all be found under https://www.gstatic.com/ct/log_list/v2/

Specifically, the following are now available:

https://www.gstatic.com/ct/log_list/v2/all_logs_list.json - the log list containing all known Logs (past and present).
https://www.gstatic.com/ct/log_list/v2/log_list.json - the list of Logs that are currently compliance with Chrome's CT Log policy (or have been and were retires), and are included in Chrome.
https://www.gstatic.com/ct/log_list/v2/log_list.sig - Google's signature for log_list.json
https://www.gstatic.com/ct/log_list/v2/log_list_pubkey.pem - the public key required to verify log_list.sig

We will be updating the page at https://www.certificate-transparency.org/known-logs shortly to reflect this.

Some of you will be aware that we were serving these lists from a v2_beta directory previous to this announcement.  This is notice that on September 30th 2019 https://www.gstatic.com/ct/log_list/v2_beta/* will be removed.  Therefore, you should switch over to using the lists hosted at https://www.gstatic.com/ct/log_list/v2/ as soon as possible.

Do let us know if you find any issues with these new lists!  Otherwise, we hope they are of value to you!

Kat

To unsubscribe from this group and stop receiving emails from it, send an email to certificate-transparency+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "certificate-transparency" group.
To unsubscribe from this group and stop receiving emails from it, send an email to certificate-transparency+unsub...@googlegroups.com.

Kat Joyce

unread,
Aug 22, 2019, 10:06:22 AM8/22/19
to certificate-...@googlegroups.com
Hi,

Yes we are still currently supporting the version one Log list - although we will be deprecating that at some point too, so it might be worth switching to the new one at some point soon.  However, we will make an announcement with plenty of advance notice of when we plan to turn that down.

So the v1 signature should be verifying!  I'll look in to this for you and see what's going on :)

Kat

To unsubscribe from this group and stop receiving emails from it, send an email to certificate-transp...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "certificate-transparency" group.
To unsubscribe from this group and stop receiving emails from it, send an email to certificate-transp...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "certificate-transparency" group.
To unsubscribe from this group and stop receiving emails from it, send an email to certificate-transp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/certificate-transparency/9316f0c1-d5e8-44cb-8e5b-6814bf7a5f1f%40googlegroups.com.

Kat Joyce

unread,
Aug 22, 2019, 10:43:49 AM8/22/19
to certificate-...@googlegroups.com
Hi again,

I seem to be unable to reproduce the issue you're having - I've just downloaded the various files and have managed to verify the signature.

Is there any further information you can provide to help work out this issue?  Could you attach the exact versions of each of the files you're verifying against?

Has anyone else on this mailing list been seeing problems when trying to verify the signature of the log list?

Let me know, and hopefully we can get to the bottom of this asap :)

Kind regards,
Kat
Message has been deleted

Kat Joyce

unread,
Aug 22, 2019, 11:27:36 AM8/22/19
to certificate-...@googlegroups.com
On Thu, Aug 22, 2019 at 4:07 PM SriRamji Kathiravan <sriramji....@pearson.com> wrote:
Hi Kat,

Thanks for the quick investigation we have implemented CT for our android application 

import  java.security.Signature 

 if (Signature.getInstance("SHA256WithRSA").apply {
                    initVerify(publicKey)
                    update(message.toByteArray())
                }.verify(signature)) {
                LogServerSignatureResult.Valid
            } else {
                LogServerSignatureResult.Invalid.SignatureFailed
            }


Signature verification constantly failed today(same code works fine until yesterday) and return LogServerSignatureResult.Invalid.SignatureFailed

Note : If i change the message  & signature value to version 2 URL value it gives me the valid result

This is to be expected, as we use the same key pair for signing both the v1 and v2 lists.  But obviously it should therefore verify for both message/signature pairs!

We initially verified it using a short Go program, but and have now also verified it with openssl by running the following:

$ openssl dgst -sha256 -verify /tmp/log_list_pubkey.pem -signature /tmp/log_list.sig /tmp/log_list.json
Verified OK

If you try running the above commands does it work for you?
 

Al Cutter

unread,
Aug 22, 2019, 11:31:33 AM8/22/19
to certificate-...@googlegroups.com
I can verify the signature too:


$ openssl dgst -sha256 -verify log_list_pubkey.pem -signature log_list.sig log_list.json 
Verified OK

$ sha256sum  *
dff60d308221f90e85a866c2cf4622b4a8525de7e4b743bc135a3a335f3d91ef  log_list.json
a284e4ad744211a6636a03c59aca9446ecd5ff2c14d7cb6edd69f7ba2108bf95  log_list_pubkey.pem
295be25a90e3cc4ea22fd1f4f2a110027670cf3df1de1bfcb2e1e1e06b4b4aa7  log_list.sig

I'm afraid I don't really know much about Java anymore, by the looks of things it's changed a lot since I last looked at it!



On Thu, Aug 22, 2019 at 4:07 PM SriRamji Kathiravan <sriramji....@pearson.com> wrote:
Hi Kat,

Thanks for the quick investigation we have implemented CT for our android application 

import  java.security.Signature 

 if (Signature.getInstance("SHA256WithRSA").apply {
                    initVerify(publicKey)
                    update(message.toByteArray())
                }.verify(signature)) {
                LogServerSignatureResult.Valid
            } else {
                LogServerSignatureResult.Invalid.SignatureFailed
            }


Signature verification constantly failed today(same code works fine until yesterday) and return LogServerSignatureResult.Invalid.SignatureFailed

Note : If i change the message  & signature value to version 2 URL value it gives me the valid result

To view this discussion on the web visit https://groups.google.com/d/msgid/certificate-transparency/f510bea5-7d7c-49dd-a0b0-7e87fefb7892%40googlegroups.com.

Al Cutter

unread,
Aug 22, 2019, 11:37:03 AM8/22/19
to certificate-...@googlegroups.com
Incidentally, the Javadoc seems to suggest that the right string for getInstance is SHA256withRSA (fully lowercase "with") rather than SHA256WithRSA as you have in your example code.

I don't know if this is the cause or a red herring, but I thought it was worth pointing out (although I'd guess it would throw an exception of some kind if it didn't recognise the string...?)

On Thu, Aug 22, 2019 at 4:07 PM SriRamji Kathiravan <sriramji....@pearson.com> wrote:
Hi Kat,

Thanks for the quick investigation we have implemented CT for our android application 

import  java.security.Signature 

 if (Signature.getInstance("SHA256WithRSA").apply {
                    initVerify(publicKey)
                    update(message.toByteArray())
                }.verify(signature)) {
                LogServerSignatureResult.Valid
            } else {
                LogServerSignatureResult.Invalid.SignatureFailed
            }


Signature verification constantly failed today(same code works fine until yesterday) and return LogServerSignatureResult.Invalid.SignatureFailed

Note : If i change the message  & signature value to version 2 URL value it gives me the valid result

To view this discussion on the web visit https://groups.google.com/d/msgid/certificate-transparency/f510bea5-7d7c-49dd-a0b0-7e87fefb7892%40googlegroups.com.

Kathiravan, SriRamji

unread,
Aug 22, 2019, 11:51:39 AM8/22/19
to certificate-...@googlegroups.com
Hi kat,

I'm not sure we have used same code(SHA256WithRSA) for both the version.

Any way it started working for version 1 as well, thanks for the timely help.

To view this discussion on the web visit https://groups.google.com/d/msgid/certificate-transparency/CACM%3D_Ofkj_pSDqrC2xQCSUU3L_YO%2BpsmcOfUiZH6AAdOACtBNA%40mail.gmail.com.
Reply all
Reply to author
Forward
0 new messages