Turning down Google's v1, v2 CT log list publishing

[Devon O'Brien]

Hello ct-policy@,

In November 2021, we announced some changes to Chrome’s CT log list schema, moving from v2 to v3 in order to accommodate some new Chrome-specific metadata for making CT log lists dynamically updatable. To ensure these changes didn’t cause issues with the CT ecosystem, we continued to maintain and publish changes to the older v1 and v2 schemas for several months.

We are planning to stop publishing both the v1 and v2 CT log lists on 17 October 2022. If there are any tools or other dependencies still relying on these older versions, we encourage maintainers to migrate to the v3 list before this date. While the v1 → v2 changes included a significant overhaul of the list schema, the v2 → v3 changes were minor and additive, meaning that implementations relying on v2 should be able to consume v3 log lists with little to no additional effort. Additional information can be found by referring to the log list schemas (v2, v3) themselves.

Information about the log list versions, their URLs, and planned turndown dates can be found in the following table.

Version	URL	Turndown Date
v1	https://www.gstatic.com/ct/log_list/log_list.json	2022-10-17
v2	https://www.gstatic.com/ct/log_list/v2/log_list.json	2022-10-17
v2	https://www.gstatic.com/ct/log_list/v2/all_logs_list.json	2022-10-17
v3	https://www.gstatic.com/ct/log_list/v3/log_list.json	N/A
v3	https://www.gstatic.com/ct/log_list/v3/all_logs_list.json	N/A

If there are any questions or concerns about this planned update, please reply on this thread or reach out to us at chrome-certific...@google.com so we can discuss this further.

Thanks,

-Devon

[Roger Ng]

Hello ct-policy@,

To allow for the dependencies to be updated, we are going to delay the v2 log list turndown by another month to 2022-11-17. Note that we're still planning to turn down the v1 log list on 2022-10-17.

Version	URL	Turndown Date
v1	https://www.gstatic.com/ct/log_list/log_list.json	2022-10-17

v2	https://www.gstatic.com/ct/log_list/v2/log_list.json	2022-11-17
v2	https://www.gstatic.com/ct/log_list/v2/all_logs_list.json	2022-11-17

v3	https://www.gstatic.com/ct/log_list/v3/log_list.json	N/A
v3	https://www.gstatic.com/ct/log_list/v3/all_logs_list.json	N/A

If there are any questions or concerns about this planned update, please reply on this thread for further discussion.

Thanks,
Roger

--
You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/CAD2nvsSMADWCBbYBYPD7_VRxhqm-dKe7nL7hHmD5n%2Bvepyo4DQ%40mail.gmail.com.

[Kurt Roeckx]

Moving from v2 to v3, it seems the following are missing from the
all_logs_lists.json file:
- https://ct.googleapis.com/logs/eu1/solera2024/
- https://sapling.ct.letsencrypt.org/2022h2/
- https://sapling.ct.letsencrypt.org/2023h1/


Kurt

> > <https://groups.google.com/a/chromium.org/g/ct-policy/c/IdbrdAcDQto/m/i5KPyzYwBAAJ>,

> > moving from v2 to v3 in order to accommodate some new Chrome-specific
> > metadata for making CT log lists dynamically updatable. To ensure these
> > changes didn’t cause issues with the CT ecosystem, we continued to maintain
> > and publish changes to the older v1 and v2 schemas for several months.
> >
> > We are planning to stop publishing both the v1 and v2 CT log lists on 17
> > October 2022. If there are any tools or other dependencies still relying on
> > these older versions, we encourage maintainers to migrate to the v3 list
> > before this date. While the v1 → v2 changes included a significant overhaul
> > of the list schema, the v2 → v3 changes were minor and additive, meaning
> > that implementations relying on v2 should be able to consume v3 log lists
> > with little to no additional effort. Additional information can be found by
> > referring to the log list schemas (v2

> > <https://www.gstatic.com/ct/log_list/v2/log_list_schema.json>, v3
> > <https://www.gstatic.com/ct/log_list/v3/log_list_schema.json>) themselves.

> > <https://groups.google.com/a/chromium.org/d/msgid/ct-policy/CAD2nvsSMADWCBbYBYPD7_VRxhqm-dKe7nL7hHmD5n%2Bvepyo4DQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
> > .

> >
>
> --
> You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+...@chromium.org.

> To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/CALbv53Na1E9otcK6kHjFwoD%3DzwqZrVLcpdcx7H4J58APhHrtFw%40mail.gmail.com.

[Carlos Joan Rafael Ibarra Lopez]

Hi Kurt,

Thanks for letting us know, we will add the missing logs to the v3 version of all_logs_list.json, (I've also double checked, and those are indeed the only missing ones).

-Carlos

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/Yz8zVN698SKMkOoB%40roeckx.be.

[Roger Ng]

Hello ct-policy@,

The v1 log list publishing has been turned down.

Thanks,

Roger

[Roger Ng]

Hello ct-policy@,

We have been closely monitoring the situation in the third party library (https://github.com/appmattus/certificatetransparency).

We are pleased to announce that the v2 log list endpoints will serve the v3 log list, which is backward compatible with v2, for another 90 days starting on 2022-11-17. The v2 log list endpoints will start returning 404 on 2023-02-15.

The following v2 log list endpoints will serve v3 log list data between 2022-11-17 and 2023-02-15:

• all_logs_list.json (v2 endpoint, v3 data)
• log_list.json (v2 endpoint, v3 data)
• log_list.sig (v2 endpoint, v3 data)
• log_list.zip (v2 endpoint, v3 data)
• log_list_pubkey.pem (v2 endpoint, v3 data)
• log_list_schema.json (v2 endpoint, v3 data)

If there are any questions or concerns about this, please reply to this email for further discussion.

Cheers,
Roger on behalf of Google CT Team

[Joel Oughton-Estruch]

Hello,

** URGENT **

We missed this announcement and this change has caused SSL failures across all our Android Apps on end user devices.

Can the V2 endpoints (e.g., https://www.gstatic.com/ct/log_list/v2/log_list.json) please be restored to redirect to the V3 endpoints until all App like ours can be given enough time to update and end users update on their devices.

I'm sure there are many other Apps that are failing due to the removal of the v2 endpoints.

Thanks,

Joel Oughton-Estruch

Engineering Manager | TrueLayer

[Saumya Singh Rathore]

We are using appmatus lib for certificate transparency. And our network traffic is not able to connect to our servers due to the v2 log issue which expired on 15 Feb 23. Requesting if can we extend this 90 days valid time to one more day or if is there any other way to start the flow of traffic using the v2 log list. We are India’s largest gaming company WinZO. Can you please help us on the same matter? Highly appreciated.

The impact on our business is huge. We have off-the-deck/playstore distribution through our website www.winzogames.com. We have 100 million registered users and this transition would require us to float a new APK/ force update. As you would know there is a significant funnel drop. Our app-only business is down for the last 2 hours and we are losing significant traffic every second. 

Kindly help us. 

Requesting for a quick response as its killing our business.

 

Thanking much in anticipation.

[Parth Gupta]

Hi

We are using appmattus library for Certificate Transparency, causing all our APIs to fail.
Library is internally using V2 endpoint  https://www.gstatic.com/ct/log_list/v2/log_list.json, can you please redirect all V2 calls to V3 for short period of time 

Thanks 
   

[Anshul Agarwal]

** URGENT ** 

Dear Roger,

We connect on LinkedIn.  our network traffic is not able to connect to our servers due to the v2 log issue which expired on 15 Feb 23. Requesting if can we extend this 90 days valid time to one more day or if is there any other way to start the flow of traffic using the v2 log list. 

We are India’s largest gaming company WinZO. Can you please help us on the same matter? Highly appreciated.The impact on our business is huge. We have off-the-deck/playstore distribution through our website www.winzogames.com.

We have 100 million registered users and this transition would require us to float a new APK/ force update. As you would know there is a significant funnel drop. Our app-only business is down for the last 2 hours and we are losing significant traffic every second.Kindly help us.Requesting for a quick response.Thanking much in anticipation.

[Oshin Gupta]

Hi Roger,

We are using appmatus lib for certificate transparency. And our network traffic is not able to connect our servers due to v2 log issue which is expired on 15 feb 23.
I just want to know can we extend this 90 days valid time to  one more day or is there any other way to start the flow of traffic using v2 log list. We are India’s largest gaming company WinZO. Can you please help us on the same matter. 

We connected on linkedIn as well regarding the same. Please help.

Thanks

[Puneet Sharma]

** URGENT **

Can we please revert this back and give some more period for the fix ? We got impacted due to this and I am sure there will be other apps too.

[Ramakrishna reddy]

Hi Roger/Devon/Kurt

--VERY URGENT --

We are using the Babylon health library for Certificate Transparency, causing all our APIs to fail. It is impacting our apps which have millions of user bases. 
Could you please redirect calls for short period time so that we can update apps to Playstore. 

Thanks 
Ram

[Ramakrishna reddy]

We are using V1 log list ( https://www.gstatic.com/ct/log_list/log_list.json ) . Could you please redirect from V1 to V3 at-least for 1 week. We have 16 android apps need to publish to play store.

Thanks.

[Dusan Susic]

Our applications are impacted and unusable, too.

It would be great to revert this change and give extra time to make changes on app side. 

[Saumya Singh Rathore]

We are 100 million registered users across 3 apps on Playstore and off-playstore both. We are down for last 3-4 hours and completely out of business with a massive funnel drop on users. Request for time extension on this please. This is a huge business impact - urge for kind consideration. 

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/e87cc120-ef91-41b0-a945-70299f0d422cn%40chromium.org.

[Saumya Singh Rathore]

We are leaning on Google Team members on this thread to please help with potential resolutions and timelines towards resolution to help save our business and contain ongoing losses.

[Vishaal Vadher]

Hi CT team 

I work for a company called Fairmoney we are a digital bank with millions of users in Nigeria and we seem to be caught up in this certificate transparency v3 upgrade, as we are an app only bank we are super impacted by this change and so are our users who can not access their services. We would be so grateful if the v2 could be brought back to life while we upgrade to v3. As you can imagine the impact for us and our users is massive. 

Regards, 

Vishaal 

[vivek jha]

Hi Guys, 

if you were using the Babylon Health library for CT implementation. You may use this fork for v3 log list file

https://jitpack.io/#vivekkjha/certificate-transparency-android/v1

dependencies {     implementation 'com.github.vivekkjha.certificate-transparency-android:certificatetransparency:v1'    }

[Roger Ng]

Hello ct-policy@,

A rollback is underway and will land within the next few hours.

Cheers,
Roger on behalf of Google CT Team

[Saumya Singh Rathore]

🥳🥳🥳🥳🥳

WinZO Team loves you. 

P.S. can the hours be converted to minutes 🥹🙏🏼🙈

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/de685d8d-225f-4bb4-a2b3-1d4a5863dc83n%40chromium.org.

[Anshul Agarwal]

Thanks Google team,

Working now. 

Regards,

Anshul Agarwal

[Saumya Singh Rathore]

V2 is back 🚀🚀🥁🥁🙏🏼🙏🏼

On Wed, 15 Feb 2023 at 9:22 PM, Anshul Agarwal <ans...@winzogames.com> wrote:

Hi Roger,

Thank you for supporting us in this crucial hours. How long this activity take time at your ends? As we are getting hundreds for message on social media from our users. These user has real money wallet attach to app. 

Regards,

Anshul Agarwal

On Wednesday, February 15, 2023 at 8:52:36 PM UTC+5:30 Saumya Singh Rathore wrote:

🥳🥳🥳🥳🥳

You received this message because you are subscribed to a topic in the Google Groups "Certificate Transparency Policy" group.
To unsubscribe from this topic, visit https://groups.google.com/a/chromium.org/d/topic/ct-policy/zejEtWAJtEA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ct-policy+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/bc0a820e-c22b-4ce9-b4cf-4ed3f719f901n%40chromium.org.

[Udi Ben Senior]

Thank you very much!

[Seo Suchan]

what a scream test! if Google still wants to retire v2 endpoint it may assign a brownout date(turn endpoint off for a day) so if there still people using it?

2023년 2월 16일 목요일 오전 1시 23분 4초 UTC+9에 Udi Ben Senior님이 작성:

[Mike Strat]

Hello,

thanks for the rollback.

is there a deprecation date for v2 (post rollback)?

[Gal Cohen]

Thanks for the rollback, can you please update once schedule for V2 deprecation is set?

ב-יום חמישי, 16 בפברואר 2023 בשעה 10:17:37 UTC+2, Mike Strat כתב/ה:

[Roger Ng]

Hi folks,

Sorry about the continued excitement on the turndown of the v2 log list. We're still exploring what we should do going forward, and we'll follow up when we know more, but I wanted to share a few key points:

Firstly, as noted above, the v2 log list URL is back up for now. We won't turn it back down without giving plenty of notice on this forum.

Secondly, we do still intend on turning down the v2 list. We strongly recommend relying applications implement workarounds now. Don't wait until the list goes down again!

CT enforcement is, first and foremost, there to keep the web safe. To do that, we may need to make changes to Chrome's CT implementation and policies even if those changes aren't expected by non-Chrome apps. Apps or libraries enforcing CT, especially those relying on Chrome's CT resources, need to actively monitor and adapt with these ecosystem changes to ensure their compatibility. Failure to do so will always risk breakage, and breakage can occur even with regular updates to our log lists, not just with big changes like this list URL change.

CT offers valuable security properties for apps, but unfortunately, none of the existing CT enforcement libraries we're aware of consistently keep up with ecosystem changes. This isn't a great situation, and we're exploring whether Google can help in this area. We may have more to share in the future.

As always, I'm happy to answer any questions,
Roger on behalf of the Google CT and Chrome Teams

[Paul Staden]

HI Roger, 

Just a query, this change impacted our company also. We didnt get any email notification (we have searched everywhere) Is there a way we can be added to a "notifications list" if we provide an email address? 

Thanks 

Paul 

[paul....@bt.com]

Hi,

 

Im thinking more proactively so we are aware of a change that is going to be put in. For example, in future if something else impacts our applications, is there a way we can be notified before a change starts? If I subscribe to this thread it would not tell me of anything else that’s upcoming regarding changes or alterations to google.

 

Regards

 

Paul

 

From: Denzil Ferreira <ferreir...@gmail.com>
Sent: 23 February 2023 09:09
To: Staden,P,Paul,QCB5 R <paul....@bt.com>
Cc: Certificate Transparency Policy <ct-p...@chromium.org>; Roger Ng <rog...@google.com>; Devon O'Brien <asymm...@google.com>; Kurt Roeckx <ku...@roeckx.be>
Subject: Re: [ct-policy] Turning down Google's v1, v2 CT log list publishing

 

You don't often get email from ferreir...@gmail.com. Learn why this is important

You  can subscribe to this message thread?

 

D

--
You received this message because you are subscribed to a topic in the Google Groups "Certificate Transparency Policy" group.
To unsubscribe from this topic, visit https://groups.google.com/a/chromium.org/d/topic/ct-policy/zejEtWAJtEA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ct-policy+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/d9e95d71-2e8b-4e2b-91e3-93afe79dac9dn%40chromium.org.

 

[Chandan Shivhare]

Hi All, Received below communication from Google :

Hello,

Thank you all for your interest in using Certificate Transparency. We understand the disturbance that the recent turndown caused. As we have all seen over the last few days, client-side CT enforcement comes with technical challenges and risks. We are working to provide a better experience in the future. Today, we have two announcements:

Firstly, https://www.gstatic.com/ct/log_list/v2/log_list.json will start returning 404 in 90 days, on 2023-06-07 around 10AM UTC+1.

Secondly, the existing log lists represent Chrome’s up to date interpretation of the CT ecosystem and are intended for use by CAs and CT monitors, not for CT enforcement by clients. The priority for these lists is to protect Chrome’s users.

We ask that you do not rely on these log lists for your application's CT enforcement. If you choose to ignore this warning, please:

1. Understand that this list may change without notice. This might break your application, and you must be prepared to remedy that breakage without assistance from Google.

2. Commit to monitoring the ct-p...@chromium.org mailing list and updating your application as needed. This mailing list is the official communication channel for log list changes, and we try to announce upcoming changes there when possible.

3. Understand that many existing CT enforcement libraries are not maintained, and do not follow current best practices. This may increase your app's risk of breakage.

4. Ensure that downtimes of these log lists do not result in your service incurring an outage.

We acknowledge that client-side CT enforcement on Android requires a lot of effort today. We are exploring options to make client-side CT enforcement easier and frictionless on Android. Google’s CT team will share information about these plans on the certificate-transparency mailing list in the coming months.

We remain available on certificate-...@googlegroups.com for any followup questions.

Sincerely,

Google’s CT and Chrome teams